I have annex with a subdir documents/ where I set numcopies to 4 using .gitattributes. I have a bunch of special remotes in an archive group with (not (copies=archive:2)) or lackingcopies=1 as preferred content expression.
Now, this directory has over 1000 documents in it. When I run git annex sync --content, git annex will try to drop 1000 documents from 5 archive remotes (one by one) since the files are not wanted (copies > 2) BUT it will conclude it can't because I need at least 4 copies.
This creates incredible amount of churn to the point remotes start to rate-limit me because it takes hours to resolve a single sync.
I tried to mess around with include and exclude but I can't make it work. Is this even possible? For example, I tried
(include="documents/*" and (not (copies=archive:4))) or (exclude="documents/*" and (not (copies=archive:2)))
Now, I would interpret this as "all the files in documents/ with less than 4 copies in archive OR all the files not in documents/ and less than two copies". But with this expression (and many similar variants) it always tries to drop the 4+ copy files.
Thanks!
You are not supposed to quote the pattern after 'include='
So, what your expression actually says is if there is a directory named
"documents/that contains files ending in", to applynot (copies=archive:4).I am kind of surprised that the original situation would lead to any churn though. It seems like, since it knows there are at most 3, copies, and 4 copies are required, it should be able to skip trying to drop at all. Instead it does unnecessary work. Filed a bug, unnecessary work when drop cannot possibly succeed.
I have not been able to reproduce that. There is code that should prevent
git-annex sync --contentfrom doing any work when there are not enough other copies for a drop to succeed.Here is my test:
While
git-annex dropdoes do excess work in this situation, I've concluded that's ok because the user is explicitly asking it to try to drop. Butgit-annex syncdoes not seem to do excess work when the drop cannot succeed.If you can show otherwise, please do...
Hey Joey, thanks for the response. The syntax without " was a big surprise! Removing the quotes seems to make things work as expected.
Plus it seems a big problem was also an incompatibility of the rclone special remote helper and recent rclone update where they changed some output formats and the helper was reporting incorrect results. There is a bunch of forks with various fixes, it's a shame the original author disappeared.
Thank you so much for your work on git annex! Matus.