Currently git-annex migrate only hard links the objects in the local repo. This leaves other clones without the new keys' objects unless they re-download them, or unless the same migrate command is re-run, in the same tree, on each clone.

It would be good to support distributed migration, so that whatever migration is done in one repo is reflected in the other repos.

This needs some way to store, in the git repo, a mapping between the old key and the new key it has been migrated to. (I investigated how much space that would need in the git repo, in this comment.) The mapping might be communicated via the git branch but be locally stored in a sqlite database to make querying it fast.

Once that mapping is available, one simple way to use it would be a git-annex command that updates the local repo to reflect migrations that have happened elsewhere. It would not touch the HEAD branch, but would just hard link object files from the old to new key, and update the location log for the new key to indicate the content is present in the repo. This command could be something like git-annex migrate --update.

Rather than a dedicated command that users need to remember to run, distributed migration could be done automatically when merging a git-annex branch that adds migration information. Just hardlink object files and update the location log.

It would be possible to avoid updating the location log, but then all location log queries would have to check the migration mapping. It would be hard to make that fast enough. Consider git-annex find --in foo, which queries the location log for each file.

--Joey

done --Joey

security

It is possible for bad migration information to be recorded in the git-annex branch by someone malicious. To avoid bad or insecure behavior when bad migration information is recorded:

• When updating the local repository with a migration, verify that the object file hashes to the new key before hardlinking.

This was done.

That leaves at these possible security problems:

• DOS by flooding the git-annex branch with migrations, resulting in lots of hard links (or copies on filesystems not supporting hard links) and hashing of large files.

Note that a malicious person who can write to the git-annex branch can already set their own repo as trusted, wait for someone to drop their local copy, and then demand a ransom for the content. For that matter, someone hosting a git-annex remote on a server can wait for someone to rely on it to contain the only copy of content and ransom it then.

git-annex is probably not normally used in situations where we need to worry about this kind of attack; if we don't trust someone we shouldn't pull the git-annex branch from them, and should not trust their remote to contain the only copy.

If we pull a git-annex branch from someone, they can already DOS disk space and CPU by checking a lot of junk into git. So maybe a DOS by migration is not really a concern.

If people are worried about this kind of thing, they can avoid using the feature. --Joey