Consider if file foo uses key K, and file archive/bar uses the same key K.
Using standard client preferred content settings, git annex drop --auto
will want to drop archive/bar, but git annex get --auto will want to get
foo. git annex sync --content will do both operations, getting and then
dropping the key. Running these commands repeatedly churns unncessarily.
In the preferred content expressions for standard groups, the only place this bug can be triggered involves archive directories of repositories in the client group. A file both in the archive directory and in another directory has indeterminite status.
Fixing this needs a map from key to files using it. Then, when checking
preferred content of archive/bar, it can see that foo also uses the
key. Since foo is wanted, it should not drop the key, even though
archive/bar is not wanted.
Such a map exists, in the keys database, but only in v7 mode repositories. So, this seems solvable in v7 repositories, but not in v5. Also, the associated files map may not be accurate at all times, so that's a wrinkle to using it for this. Also, only unlocked files get into the associated files map. --Joey
I ran into this exact situation trying to split content across multiple repos by directory, and ran into a couple of files with many duplicate keys throughout the repo. I worked around it by applying a tag to the offending key, and including that tag as part of the preferred content expression:
This would result in a file possibly being included in a repo where it wouldn't be otherwise (if it wasn't in one of the include directories to begin with), but in my case, the common files were few and small (2, both less than 100K).
See also: https://git-annex.branchable.com/todo/option_to___96__drop_path__96___to_not_drop___34__all_copies__34__/ Which needs the same information.
Occurs to me one way to make sure git-annex has the information up-to-date would be to stat the index file, and if it's changed since last time, use
git ls-files --cachedto find all annexed files and update the keys database. But this could be quite slow.git-diff/git-diff-index/git-statusto update just the keys that changed?_path. Supposeincludeifany=globwas true for a key if any value in_pathmatchedglob, whileincludeifall=globwas true if all values in_pathdid. Treatinclude=globasincludeifany=globandexclude=globasnot includeifall=glob. Theninclude=* and exclude=archive/*unambiguously means "include all keys except those used only underarchive/". (This has the problem thatinclude=globis not the same asnot exclude=glob. But the meaning seems to match the typical current usage ofincludeandexclude?)I think Ilya is on the right track with recording the last treeish scanned for keys and diffing to the current treeish. This seems worth implementing.
So, plan:
git write-tree, and when it differs from the last recorded such treegit diff-index --cachedwith the last recorded tree, parse and cat the links, and update the keys database associated files with the keys it found. Note that it can also remove associated files when files get deleted, which is currently not done.archive/barbecausefoois still preferred content.This would not notice if a file was moved or deleted but that was not staged in the index yet. I think that's ok -- If the file is moved and not staged yet, it's not been added to git, so it's acceptable for git-annex to not try to keep content around for it (git-annex unused would not either in similar situations). If the file is deleted, git-annex might not drop the content yet because it's still present in the index, but the same happens if you delete a file and then
git annex drop ., because it skips over the deleted file whether or not it's staged in the index.It seems sufficient to only do this when dropping a file, not in other commands. That means the associated files database remains incomplete for other commands, not including recent changes to locked files. Since this is only a problem for dropping, and the associated files database is not needed for locked files generally, that seems ok, although it would certianly be a nice simplification if the database was always kept up-to-date. Since the update could be a pretty expensive operation, probably best to keep it for the drops that really need it.
Note that the
git write-treewould mean that using git-annex could create these tree objects, which never end up getting used. gc would eventually delete them, but there might be a situation where doing this a lot bloats the repo.There is overlap in the above plan with Database.Keys.reconcileStaged. That already stats the index, and updates associated files for changes made in the index. It does not handle locked files, and it only diffs from the index to HEAD, so when git checkout is used, it does not see those changes. (git runs the git-annex clean filter for those, and that updates the associated files database, but that also only handles unlocked files, not locked files).
reconcileStaged is run every time the keys db is opened, so it runs perhaps more frequently than is needed for this. Although it does not run in every command, or even most commands. Eg, find and whereis do not use the keys db. It might be acceptable performance to do the additional work there.
Also, handleDropsFrom queries the keys database for associated files already, and actually already checks if the key is preferred content of other files. So
git-annex sync --content, which uses that, already avoids the problem for unlocked files, but not for locked files.git annex dropdoes not use that.There must also be the same problem for required content, and that should also be fixed. Will be easy to do once the information is gathered into the keys database.
Actually, in the drop --auto case, required content is automatically handled correctly, because when checking preferred content it also checks required content.
However, drop without --auto normally honors required content, but if the same content is used by 2 files, one required and the other not, dropping the latter file will drop the content of the first, violating required content settings. This seems like it ought to be fixed.
Started a branch
trackassociatedfor this. So far, I've gotten the associated files to be kept updated with changes to locked files as well as unlocked files, and when files are removed it deletes the old associated files which did not happen reliably before. Performance impact of this is TBD, but it should only slow it down in situations where a large change has been made to the index -- eg checking out a very different branch, or adding a large number of files.There is a problem though:
The old filename does not get removed in this case. What happened here is that the first git-annex add recorded the associated file itself, but no index tree was recorded yet. So when the second git-annex runs reconcileStaged, it diffs from the empty tree to the index that contains only n2. So the removal of n is not noticed.
This may not be a blocker for fixing preferred content, it just means that the list of associated files can include files that have since been deleted. The list could be double-checked once a punative second file has been found, with catKey. So it would only slow things down in the cases this bug is about and not generally. Still, it seems a shame that this case exists, because otherwise my recent changes have made old associated files get removed, and it would be nice to be able to trust that the associated files list is accurate.
Also, bloating the keys db with stale associated files could happen. Well, it could already happen, indeed it was much worse, but that was limited to unlocked files, now it can also happen for locked files.
Only fix I can think of is to make commands like git add that register an associated file to also update the cached index tree. That would let it diff from the previous index to the current one, and so notice the deletion. But it would have to be done at git-annex shutdown, and so if it were interrupted the problem could still happen. (Or, the database could have something added to it to indicate when an associated file has been added but has not been seen in the index yet, and the next update from the index could clear those flags, and remove any files still with the flag. This seems race-prone and it would need a change to the database schema.)
Also, git commit currently fails, problem with index locking inside the smudge filter, which prevents git write-tree from working. Should be fixable by detecting when the index is locked and avoiding updating then. (Fixed now.)
Thanks @joeyh for working on this.
Is the following understanding of preferred content expressions accurate:
Also, if the above if correct, is "any file path which uses the key" defined the same way as in git-annex-unused?
This is about ready to be merged, and is fully working.
Except..
git annex drop --auto --branch=fooseems likely to do something confusing, since it will check the associated files, which may not be on that branch. That combination of options doesn't really make a lot of sense anyway? Also considergit-annex drop --all --autoAhh, not a problem after all, that option combo is already disallowed!
@Ilya, it's different than git-annex unused because that looks at other branches and tags. Preferred content only relates to the currently checked out branch.
The only preferred contents expressions that this bug affects are include= and exclude= ones. For an expression like copies=1, the key's content is present or not, no matter how many files in the working tree might use that key.
Well not exactly, if foo and bar use the same content, and preferred content is set to include=foo, and I run
git annex get --auto bar, it will not get the content despite bar's key also being used by foo, which is preferred content. I only asked the command to act on bar. That is not going to change. When the content is present,git annex drop --auto barwill, once this is fixed, avoid dropping the content because it knows foo also uses it.I don't think this needs to change user's understanding of how preferred content works. It's just git-annex being smart enough to prevent a situation where repeated
git annex get --auto; git-annex drop --autowould continually do work, rather than reaching a stable point.anythingwould ensure that it gets a copy of every file ever annexed, but sounds like that might not be true.By "currently checked out branch", I meant in the repository where git-annex is running. It might be copying to a special remote or whatever, but the filename matched by include= or exclude= in a preferred content expression comes from the branch it's run in.
When run in a bare git repo (or with --all), include= and exclude= are documented not to match.