Please describe the problem.
After all the work I did here, I discvovered that I have a Git Annex-encrypted special remote that has some corrupted data on it. I can run git annex fsck --from=remotename and have Git Annex complain about all the files that have failed to fsck. But Git Annex is not updating the location log. It still thinks the data is in the special remote, after fscking it and not getting the right data out.
This makes sense if the failure was from a download corrupted in transit, but I think the files are actually corrupted in the remote. How do I make git annex fsck update the location log to say that files aren't there when they fail to fsck? Alternately, how do I get a nice list of all the filenames that failed to fsck, so I can have a script drop them from the offending remote?
What steps will reproduce the problem?
Make a Git Annex encrypted special remote.
Put some data in it.
Modify the encrypted data, to corrupt it.
Use
git annex fsckon the file you broke.See that
git annex whereisstill thinks the file is in the special remote.
What version of git-annex are you using? On what operating system?
This is Git Annex 6.20170408+gitg804f06baa-1~ndall+1 on Ubuntu 14.04.
Please provide any additional information below.
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log
$ git annex whereis info.txt
whereis info.txt (6 copies)
...
...
21a0c4ba-7255-4a9e-9baa-c638f7df68e5 -- gdrive [amazon]
...
...
...
ok
$ git annex fsck --from=amazon info.txt
...
gpg: decryption failed: bad key
2017/06/13 20:14:54 Local file system at /tmp/tmp.Cu2Dsk0jY3: Waiting for checks to finish
2017/06/13 20:14:54 Local file system at /tmp/tmp.Cu2Dsk0jY3: Waiting for transfers to finish
2017/06/13 20:14:54 Attempt 1/3 failed with 0 errors and: directory not found
2017/06/13 20:14:54 Local file system at /tmp/tmp.Cu2Dsk0jY3: Waiting for checks to finish
2017/06/13 20:14:54 Local file system at /tmp/tmp.Cu2Dsk0jY3: Waiting for transfers to finish
2017/06/13 20:14:54 Attempt 2/3 failed with 0 errors and: directory not found
2017/06/13 20:14:55 Local file system at /tmp/tmp.Cu2Dsk0jY3: Waiting for checks to finish
2017/06/13 20:14:55 Local file system at /tmp/tmp.Cu2Dsk0jY3: Waiting for transfers to finish
failed to download file from remote
2017/06/13 20:14:55 Attempt 3/3 failed with 0 errors and: directory not found
2017/06/13 20:14:55 Failed to copy: directory not found
git-annex: fsck: 1 failed
$ git annex whereis info.txt
whereis info.txt (6 copies)
...
...
21a0c4ba-7255-4a9e-9baa-c638f7df68e5 -- gdrive [amazon]
...
...
...
ok
# End of transcript or log.
Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
Yeah, it works great! If not for it I would not have noticed this data corruption until it was too late.
fsck --from remoteis supposed to update the location log when it determines that the remote does not contain the file.But in your case, the decryption failure appears to fsck as a transfer failure, which as you note can be transient. So it doesn't update the location log.
It seems that what's needed is different errors to be returned when download fails, vs when download succeeds but decryption/verification fails. Then fsck could mark the file as not being present in the remote in the latter case.
Although, that would leave the presumably corrupted encrypted data in the remote. (Unless fsck also tried to delete it.)
Also, decryption can fail for other reasons, eg missing gpg keys, and in such a case, it would be bad for fsck to decide that the remote didn't contain any content! (And super bad for it to delete it from the remote!!)
So hmm, I'm not sure about that idea.
Your idea of getting a list of files that fsck failed to download is certianly useful. Perhaps a good way would be to make
git annex fsck --from remote --jsonwork, then the json output could be parsed to get a list of files, and you could usegit annex drop --from remoteto remove the bad data. That was the easiest possible thing, so I've made that change.