I am using git annex testremote as a baseline test bench for Forgejo-aneksajo as a git-annex remote, and it is awesome to have that. I have some pain points with it though:
- I would like to use these tests to confirm that I don't accidentally give write access to read-only users. This means I would need a way to ensure that all tests which require write access fail against the remote.
- I am spawning a
git annex testremotesubprocess within the integration tests of Forgejo-aneksajo, which are written in Go. Sometimes this "large blackbox test" gets stuck in CI and I haven't figured out why yet. It would be nice to have a more transparent integration into the Forgejo-aneksajo test suite.
Both of those points could be addressed if git annex testremote provided a way to run each test individually, and to get a list of all the available tests categorized by if they are read-only or read-write. I could then integrate each as an individual sub-test into Forgejo-aneksajo's test suite and properly assert on the outcome of the test given the respective test setup.
If that's not possible for some reason it would also be an improvement with regards to the first point if there was something like a git annex testremote --write-only with the option to only report success if all of those tests have failed.
What do you think?
It would be possible to make
git-annex testremotesupport the command-line options of the underlying test framework (tasty).git-annex testalready does that, so has --list-test and --pattern.It's not as simple as just plumbing that up though, because testremote has implicit dependencies in its test ordering. It has to do the
storeKeytest before it can do thepresenttest, for example. Those dependencies would need to be made explict, rather than implicit.Explict dependencies, though, would also make it not really possible to run most of the tests separately. Running testremote 5 times to run the listed tests, if each run does the necessary
storeKeywould add a lot of overhead.Not declaring dependencies and leaving it up to the user to run testremote repeatedly to run a sequence of tests in the necessary order would also run into problems with testremote using random test keys which change every time it's run, as well as it having an end cleanup stage where it removes any lingering test keys from the local repository and the remote.
This seems to be a bit of an impasse...
I don't know about the "--write-only" name, but I see the value in having a way for testremote to check what a remote that is expected to only allow read access does not allow any writes, as well as otherwise behaving correctly.
There are actually only two write operations,
storeKeyandremoveKey. SinceremoveKeyis supposed to succeed when a key is not present, ifstoreKeyfails, thenremoveKeywill succeed. ButremoveKeyshould fail to remove a key that is stored on the remote. To test that, the --test-readonly=file option would need to be used to provide a file that is already stored on the remote.I think it would make sense to require that option be present in order to use this new "--write-only" (or whatever name) option.
Also, git-annex does know internally that some remotes are readonly. For example, a regular http git remote that does not use p2phttp. Or any remote that has
remote.<name>.annex-readonlyset. Currentlytestremoteonly skips all the write tests for those, rather than confirming that writes fail. It would make sense for testremote of a known readonly remote to behave as if this new option were provided.(But, setting
remote.<name>.annex-readonlyrather than using the "--write-only" option would not work for you, because that config causes git-annex to refuse to try to write to the remote. Which doesn't tell you if your server is configured to correctly reject writes.)I already thought that this might be the case, so running the tests independently isn't really infeasible.
To address my second point I might be able to just parse the output of testremote into "sub-tests" on the Forgejo-aneksajo side. Tasty doesn't seem to have a nice streaming output format for that though, right? There is a TAP formatter, but that looks unmaintained...
Now that you are saying this, is a new option even necessary? --test-readonly already takes a filename that is expected to be present on the remote, so instead of adding a new option --test-readonly could ensure that this key can't be removed, and that a different key can't be stored (and that removeKey succeeds on this not-present key).