Some time ago I asked here
about possible improvements in git
copy --fast --to, since it was painfully slow
on moderately large repos.
Now I found a way to make it much faster for my particular use case, by
accessing some annex internals. And I realized that maybe commands like
annex find --in=repo do not batch queries to the location log. This is based on
the following timings, on a new repo (only a few commits) and about 30k files.
> time git annex find --in=skynet > /dev/null real 0m55.838s user 0m30.000s sys 0m1.583s > time git ls-tree -r git-annex | cut -d ' ' -f 3 | cut -f 1 | git cat-file --batch > /dev/null real 0m0.334s user 0m0.517s sys 0m0.030s
Those numbers are on linux (with an already warm file cache) and an ext4 filesystem on a SSD.
The second command above is feeding a list of objects to a single
process that cats them all to stdout, preceeding every file dump by the object
being cat-ed. It is a trivial matter to parse this output and use it for
whatever annex needs.
Above I wrote a
git ls-tree on the git-annex branch for simplicity, but we could
just as well do a
ls-tree | ... | git cat-file on HEAD to get the keys for the
annexed files matching some path and then feed those keys to a cat-file on
the git-annex branch. And this still would be an order of magnitude faster than
what currently annex seems to do.
I'm assuming the bottleneck is in that annex does not batch the
cat-file, as the rest of logic needed for a find will be fast. Is that right?
Now, if the queries to the location log for
copy --to and
find could be batched this way, the
performance of several useful things to do, like checking how many annexed files
we are missing, would be bastly improved. Hell, I could even put that number on
the command line prompt!
I'm not yet very fluent in Haskell, but I'm willing to help if this is something that makes sense and can be done.