Say we have 2 backup drives and want to fill them both evenly with files, different files in each drive. Currently, preferred content cannot express that entirely:

  • One way is to use a-m and n-z, but that's unlikely to split filenames evenly.
  • Or, can let both repos take whatever files, perhaps at random, that the other repo is not know to contain, but then repos will race and both get the same file, or similarly if they are not communicating frequently.

So, let's add a new expression: balanced_amoung(group)

This would work by taking the list of uuids of all repositories in the group, and sorting them, which yields a list from 0..M-1 repositories.

To decide which repository wants key K, convert K to a number N in some stable way and then N mod M yields the number of the repository that wants it, while all the rest don't.

(Since git-annex keys can be pretty long and not all of them are random hashes, let's md5sum the key and then use the md5 as a number.)

This expression is stable as long as the members of the group don't change. I think that's stable enough to work as a preferred content expression.

Now, you may want to be able to add a third repo and have the data be rebalanced, with some moving to it. And that would happen. However, as this scheme stands, it's equally likely that adding repo3 will make repo1 and repo2 want to swap files between them. So, we'll want to add some precautions to avoid a lof of data moving around in this case:

((balanced_amoung(backup) and not (copies=backup:1)) or present

So once file lands on a backup drive, it stays there, even if more backup drives change the balancing.

Some limitations:

  • The item size is not taken into account. One repo could end up with a much larger item or items and so fill up faster. And the other repo wouldn't then notice it was full and take up some slack.
  • With the complicated expression above, adding a new repo when one is full would not necessarily result in new files going to one of the 2 repos that still have space. Some items would end up going to the full repo.

These can be dealt with by noticing when a repo is full and moving some of it's files (any will do) to other repos in its group. I don't see a way to make preferred content express that movement though; it would need to be a manual/scripted process.

What if we have 5 backup repos and want each file to land in 3 of them? There's a simple change that can support that: balanced_amoung(group:3)

This works the same as before, but rather than just N mod M, take N+I mod M where I is [0..2] to get the list of 3 repositories that want a key.

This does not really avoid the limitations above, but having more repos that want each file will reduce the chances that no repo will be able to take a given file. In the iabackup scenario, new clients will just be assigned until all the files reach the desired level or replication.

However.. Imagine there are 9 repos, all full, and some files have not reached desired level of replication. Seems like adding 1 more repo will make only 3 in 10 files be wanted by that new repo. Even if the repo has space for all the files, it won't be sufficient, and more repos would need to be added.

One way to avoid this problem would be if the preferred content was only used for the initial distribution of files to a repo. If the repo has gotten all the files it wants, it could make a second pass and opportunistically get files it doesn't want but that it has space for and that don't have enough copies yet. Although this gets back to the original problem of multiple repos racing downloads and files getting more than the desired number of copies.