Please describe the problem.
I'm migrating certain files from SHA256E to SHA256, to avoid problems caused by inconsistent-case extensions, and by weird filenames which git-annex misdetects as part of the extension. The obvious approach, annex migrate --to SHA256, does not work as expected – the new keys still have the same extension appended. This kinda defeats the point.
I can work around this by migrating to another hash first (e.g. to SHA1), then back.
What steps will reproduce the problem?
- Have a SHA256E or SHA1E file
- git annex migrate --backend=SHA256
What version of git-annex are you using? On what operating system?
6.20180509-g0632c49c22, Linux
Please provide any additional information below.
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log
$ ls -l
lrwxrwxrwx 1 grawity grawity 211 May 2 07:58 EN_EXCH2003_ENT.ISO -> ../../../.git/annex/objects/41/Fq/SHA256E-s391118848--3b0a03ce8a821c98de7d3e67f9664f55b4bb7e13855721db993881bad501caf3.ISO/SHA256E-s391118848--3b0a03ce8a821c98de7d3e67f9664f55b4bb7e13855721db993881bad501caf3.ISO
$ git annex migrate --backend=SHA256 --force
migrate EN_EXCH2003_ENT.ISO (checksum...) ok
(recording state in git...)
$ ls -l
lrwxrwxrwx 1 grawity grawity 209 May 22 08:15 EN_EXCH2003_ENT.ISO -> ../../../.git/annex/objects/09/3P/SHA256-s391118848--3b0a03ce8a821c98de7d3e67f9664f55b4bb7e13855721db993881bad501caf3.ISO/SHA256-s391118848--3b0a03ce8a821c98de7d3e67f9664f55b4bb7e13855721db993881bad501caf3.ISO
$ git annex migrate --backend=SHA1
migrate EN_EXCH2003_ENT.ISO (checksum...) (checksum...) ok
(recording state in git...)
$ git annex migrate --backend=SHA256
migrate EN_EXCH2003_ENT.ISO (checksum...) (checksum...) ok
(recording state in git...)
$ ls -l
lrwxrwxrwx 1 grawity grawity 201 May 22 08:17 EN_EXCH2003_ENT.ISO -> ../../../.git/annex/objects/xX/GW/SHA256-s391118848--3b0a03ce8a821c98de7d3e67f9664f55b4bb7e13855721db993881bad501caf3/SHA256-s391118848--3b0a03ce8a821c98de7d3e67f9664f55b4bb7e13855721db993881bad501caf3
# End of transcript or log.
I suspect this is a bug in the fastMigrate code.
Interestingly, the SHA256 key with an exctension works ok, eg fsck can verify the checksum.
Also, a subsequent
git annex migrate --backend=SHA256Eresults in a SHA256E key with no extension.The bug was that it got the test backwards for whether it was supposed to be adding or removing the extension!
Bug was introduced in 9c4650358ca85a298b747bb897dbf4f8f891fa22 over a year ago.
The bogus SHA256 key with an extension tacked on at the end passes fsck because the code happens to always strip extensions from hashes, even if the key type is not supposed to include an extension.
Fixed the bug. But this leaves the potential for these badly formed SHA256 keys with an extension on the end being in a repository and needing to keep code working for them. (The SHA256E keys without an extension that also result from the bug are technically not badly formed.)
So, I also made migrate fix those badly formed keys. You have to specify --backend=SHA256, and then it will migrate the badly formed SHA256 key to a correctly formed SHA256 key.
Also, git-annex fsck will now warn when it detects a key needing such a migration.