If a S3 remote is set up with exporttree=yes, and some files are stored on it, and then it's later changed to also have versioning=yes, an exporttree that removes some of the original files can lose the only remaining copy of them.

An appendonly remote, such as S3 with exporttree=yes, is supposed to not let git-annex remove content from it. So such a remote can be not untrusted, and exporttree can remove content from its exported tree without violating numcopies since the content is still supposed to be available in the remote.

The S3 remote that gets versioning=yes enabled after some content has been stored on it without versioning violates the requirements for an appendonly remote. When exporttree removes a file from that S3 remote, it could have contained the only copy of the file, and it may not have versioning info for that file, so the only copy is lost.

Migration advice for users affected by this bug

If you think your S3 remote may be affected by this problem, you should immediately set it to untrusted to avoid data loss: git annex untrust $mys3remotename

If you see a warning message "Remote is configured to use versioning, but no S3 version ID is recorded for this key", your S3 remote is affected.

Also, the fixed git-annex (version 7.20190129) will detect the problem, and refuse to delete unversioned files from your versioned S3 bucket.

This will leave you with a S3 remote containing some versioned and some unversioned files. Kind of a mess. Best thing to do is to make a new S3 remote, with versioning=yes exporttree=yes set from the beginning, and copy all the content that was in the old S3 remote over to it. Then you can delete the old S3 bucket, and use git annex dead to make git-annex stop using it.

Fix

  • Auto-enable versioning during initremote (and not enableremote) when versioning=yes. (Or prompt user to do it when aws is too old.)
  • Do not allow changing versioning= during enableremote.
  • Make removeExport and renameExport check that there is a S3 version ID known, and fail if not.

done --Joey