v9 is implemented, and enables filter-process. Not yet the default, but will be eventually.

v10 is implemented, so upgrade your affected repositories to it to get the fix.

What kind of locking is needed for the v10 upgrade? Everything else is in place now, but the actual locking is TBD.

My plan above calls for a way to detect if another git-annex process (v9 or above) is running in the repo. That will be hard to implement though..

It might cause problems with annex.pidlock, if every git-annex process starts taking a shared lock, because pidlock does not support shared locks, so only 1 git-annex process will be able to run, perhaps in situations where multiples can run now even with pidlocking because no locking is needed.

Also, the existing locking machinery runs in the Annex monad, but such a lock needs to be implemented in Annex.hs itself, so that would be a recursive dep. And, it would add overhead to every git-annex process. (A small amount.)

Alternatively, there could be a top-level lock file that is held shared whenever locking content files. And the v10 upgrade takes an exclusive lock. But this seems to fail when a v9 process is running -- if it blocks on the shared lock for the v10 upgrade, it would still go on the lock in v9 mode in the now v10 repository.

Update: That problem can be avoided by re-reading the git config to check if v10 was enabled, once it has taken the shared lock. That will mean that v9 repos do a little bit more work when locking content and dropping. For efficiency, use an InodeCache and only re-read when it's changed. This will need the v10 upgrade to set annex.version while it is still holding the exclusive lock.

I agree, submodules are the usual way to nest git repositories, and will more or less just work with git-annex.

I think that the author of this tip is wanting to version control the contents of .git itself. Eg, to version control .git/config and .git/hooks/.

One problem with this approach is that when the outer repository has "dotgit/annex/objects/files added to it, runninggit-annex dropinside the nested git repository will drop the content, but the outer repository will still contain a copy too. You would have to usegit-annex unused` to eventually clean up those copies. And it stores 2 copies of every annexed file to use it this way.

After running a git status today (the first time running it was slow as usual) the problem seems to have gone away. Perhaps it just needed to rebuild some kind of cache or something. I believe that I had waited for a git status command to completely finish before, so maybe it takes several tries to resolve this performance problem? Or possibly I'm misremembering and I never actually waited for the git status command to fully finish. I'm baffled by this, but happy that it's back to normal. I'll post an update if the problem comes back.

Using nested git repositories is well possible; if they are checked in they are called submodules, otherwise they just sit there unadded.

Apart from some odd quirx you never run into in normal operation, submodules work fine also with git-annex.

The main problem is that the git developers stubbornly refuse to support nested git repos, so it's impossible for git-annex to support it. However I found a good workaround, see using nested git repositories.

I found an even better workaround, see using nested git repositories

This is going to need two repository version bumps:

v9: Add the upgrade lock file, and all git-annex processes take a shared lock to avoid the repository being upgraded out from under them.

v10: Skipped until the upgrade lock file is of a certain age. Take upgrade lock before upgrading.

In v10, stop locking content files and lock separate lock files.

The age could be eg 1 month, which assumes that no pre-v9 git-annex process like git-annex move --to remote is still running after that long. Of course, that is still an assumption, but it can be pushed out as long as it takes to feel comfortable with it. Maybe 1 year? The only disadvantage really is that any v11 upgrade would also get deferred.

Since the assistant can possibly run for longer than a year without restarting, the v10 upgrade would need to be skipped when the assistant is running.

git-annex upgrade --version=10 could be available to speed up that upgrade. The user would be responsible for making sure there are no such old git-annex processes running, so that might need --force.

Still checking but so far looks good with 8.20211231+git69-ga55fc567c-1~ndall+1 -- thank you!