The --hide-missing and --unlock-present adjusted branches depend on whether the content of a file is present. So after a get or drop, the branch needs to be updated to reflect the change.

To avoid the user needing to do it manually, the annex.adjustedbranchrefresh config automates it now. That is not yet enabled by default, because it can be slow.

Can it be done efficiently enough to become default? And how frequently can it update without being too slow? After every file or after some number of files, or after processing all files in a (sub-)tree, or only once at the end of a command?

The config was designed as a numeric scale with the hope that it might later become a simple boolean if it got fast enough. Since 0 is false and > 0 is true, in git config, the old values will still work.

Investigation of the obvious things that make it slow follows:

efficient branch adjustment

updateAdjustedBranch re-adjusts the whole branch. That is inneficient for a branch with a large number of files in it. We need a way to incrementally adjust part of the branch.

(Or maybe not need because maybe it's acceptable for it to be slow; slow is still better than manual, probably.)

Git.Tree.adjustTree recursively lists the tree and applies the adjustment to each item in it. What's needed is a way to adjust only the subtree containing a file, and then use Git.Tree.graftTree to graft that into the existing tree. Seems quite doable!

graftTree does use getTree, so buffers the whole tree object in memory. adjustTree avoids doing that. I think it's probably ok though; git probably also buffers tree objects in memory. Only the tree objects down to the graft point need to be buffered.

Oh, also, it can cache a few tree objects to speed this up more. Eg after dropping foo/bar/1 buffer foo and foo/bar's objects, and use that when dropping foo/bar/2.

efficient checkout

updateAdjustedBranch checks out the new version of the branch.

git checkout needs to diff the old and new tree, and while git is fast, it's not magic. Consider if, after every file dropped, we checked out a new branch that did not contain that file. Dropping can be quite fast, thousands of files per second.. How much would those git checkouts slow it down?

I benchmarked to find out. (On an SSD)

for x in $(seq 1 10000); do echo $x > $x; git add $x; git commit -m add; done

time while git checkout --quiet HEAD^;do : ; done
real    5m59.489s
user    4m34.775s
sys 3m39.665s

Seems like git-annex could do the equivilant of checkout more quickly, since it knows the part of the branch that changed. Eg, git rm the file, and update the sha1 that HEAD points to.

dangling git objects

Incremental adjusting would generate new trees each time, and a new commit. If it's done a lot, those will pile up. They get deleted by gc, but normally would only be deleted after some time. Perhaps git-annex could delete them itself. (At least if they remain loose git objects; deleting them once they reach a pack file seems hard.)

(The tree object buffer mentioned above suggests the approach of, when an object in the buffer gets replaced with another object for the same tree position, delete the old object from .git/objects.)

A git repo containing 10000 commits, starting with 10000 files and ending with 0, deleting 1 file per commit, weighs in at 99mb. That's with short filenames, more typical filenames would add to it. (It packs down to 9 mb, but that's just extra work for objects that will get gc'ed away eventually.)

slowness in writing git objects

git-annex uses git commit-tree to create new commits on the adjusted branch. That is not batched, so running once per file may get slow.

And to write trees, it uses git mktree --batch. But, a new process is started each time by Git.Tree.adjustTree (and other things). Making that a long-running process would speed it up, probably.

queue flush slowness

It uses Annex.Queue.flush to avoid a problem with dropping files, which depopuates the pointer file, which git sees as modified until restaged, which then prevents checking out the version of the branch where the pointer file is a symlink. There must be a less expensive way to handle that.

(All the extra "(recording state in git...)" when dropping are due to it doing that too.)