Yesterday I spent making a release, and shopping for a new laptop, since this one is dying. (Soon I'll be able to compile git-annex fast-ish! Yay!) And thinking about ?wishlist: dropping git-annex history.

Today, I added the git annex forget command. It's currently been lightly tested, seems to work, and is living in the forget branch until I gain confidence with it. It should be perfectly safe to use, even if it's buggy, because you can use git reflog git-annex to pull out and revert to an old version of your git-annex branch. So if you're been wanting this feature, please beta test!

I actually implemented something more generic than just forgetting git history. There's now a whole mechanism for git-annex doing distributed transitions of whatever sort is needed.

There were several subtleties involved in distributed transitions:

First is how to tell when a given transition has already been done on a branch. At first I was thinking that the transition log should include the sha of the first commit on the old branch that got rewritten. However, that would mean that after a single transition had been done, every git-annex branch merge would need to look up the first commit of the current branch, to see if it's done the transition yet. That's slow! Instead, transitions are logged with a timestamp, and as long as a branch contains a transition with the same timestamp, it's been done.

A really tricky problem is what to do if the local repository has transitioned, but a remote has not, and changes keep being made to the remote. What it does so far is incorporate the changes from the remote into the index, and re-run the transition code over the whole thing to yeild a single new commit. This might not be very efficient (once I write the more full-featured transition code), but it lets the local repo keep up with what's going on in the remote, without directly merging with it (which would revert the transition). And once the remote repository has its git-annex upgraded to one that knows about transitions, it will finish up the transition on its side automatically, and the two branches will once again merge.

Related to the previous problem, we don't want to keep trying to merge from a remote branch when it's not yet transitioned. So a blacklist is used, of untransitioned commits that have already been integrated.

One really subtle thing is that when the user does a transition more complicated than git annex forget, like the git annex forget --dead that I need to implement to forget dead remotes, they're not just telling git-annex to forget whatever dead remotes it knows right now. They're actually telling git-annex to perform the transition one time on every existing clone of the repository, at some point in the future. Repositories with unfinished transitions could hang around for years, and at some future point when git-annex runs in the repository again, it would merge in the current state of the world, and re-do the transition. So you might tell it to forget dead remotes today, and then the very repository you ran that in later becomes dead, and a long-slumbering repo wakes up and forgets about the repo that started the whole process! I hope users don't find this massively confusing, but that's how the implementation works right now.

I think I have at least two more days of work to do to finish up this feature.

  • I still need to add some extra features like forgetting about dead remotes, and forgetting about keys that are no longer present on any remote.

  • After git annex forget, git annex sync will fail to push the synced/annex branch to remotes, since the branch is no longer a fast-forward of the old one. I will probably fix this by making git annex sync do a fallback push of a unique branch in this case, like the assistant already does. Although I may need to adjust that code to handle this case, too..

  • For some reason the automatic transitioning code triggers a "(recovery from race)" commit. This is certainly a bug somewhere, because you can't have a race with only 1 participant.

Today's work was sponsored by Richard Hartmann.