https://github.com/OpenNeuroOrg/openneuro/issues/3446#issuecomment-2892398583
This is a case where a truncated file was exported as part of a tree to S3.
In particular a bucket with versioning=yes.
Note that git-annex export does not verify checksums before sending, and
so it's possible for this to happen if a corrupted object has somehow
gotten into the local repository. It might be possible to improve this to
deal better with object corruption, including object corruption that occurs
while exporting.
Currently there is no good way for a user to recover from this. Exporting a tree that deletes the corrupted file, followed by a tree that adds back the right version of the file will generally work. But it will not work for a versioned S3 bucket, because removing an export from a versioned S3 bucket does not remove the recorded S3 versionId. While re-exporting the file will record the new versionId, the old one remains recorded, and when multiple versionIds are recorded for the same key, either may be used when retrieving it.
What needs to be done is to remove the old versionId. But it does not seem right to generally do this when removing an exported file from a S3 bucket, because usually, when it's not corrupted, that versionId is still valid, and can still be used to retrieve that object.
git-annex fsck --from=s3 will detect the problem, but it is unable to do
anything to resolve it, since it can only try to drop the corrupted key,
and dropping by key is not supported with an exporttree=yes remote.
Could fsck be extended to handle this? It should be possible for fsck to:
- removeExport the corrupted file, and update the export log to say that the export of the tree to the special remote is incomplete.
- Handle the special case of the versioned S3 bucket with eg, a new Remote method that is used when a key on the remote is corrupted. In the case of a versioned S3 bucket, that new method would remove the versionId.
--Joey
Note that it would also be possible for a valid object to be sent, but then get corrupted in the remote storage. I don't think that's what happened here.
If that did happen, a similar recovery process is also needed.
Which I think says that focusing on a recovery process, rather than on prevention, is more useful.
In a non-export S3 bucket with versioning, fsck also cannot recover from a corrupted object, due to the same problem with the versionId. The same method should work to handle this case.