Please describe the problem.
I'm storing lots of recordings from our satellite receiver into a git annex repo for easy syncing to a backup server that's not always switched on. Now, I'm trying to replace this backup server and pulling all data onto my new node. Most files work fine, but there are a bunch of small files that throw an error. In the debug log the GET command seems to use the file's size as offset, resulting in 0 bytes of data, throwing an error
What steps will reproduce the problem?
$ git annex get Documentaires/Oorlogen/20170124_2110_-_CANVAS_HD_-_1945__THE_SAVAGE_PEACE.* --debug
[2021-12-19 17:13:10.308043464] process [4214] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","symbolic-ref","-q","HEAD"]
[2021-12-19 17:13:10.309545353] process [4214] done ExitSuccess
[2021-12-19 17:13:10.310048003] process [4215] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","refs/heads/master"]
[2021-12-19 17:13:10.312009938] process [4215] done ExitSuccess
[2021-12-19 17:13:10.328167499] process [4216] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","ls-files","--stage","-z","--","Documentaires/Oorlogen/20170124_2110_-_CANVAS_HD_-_1945__THE_SAVAGE_PEACE.eit","Documentaires/Oorlogen/20170124_2110_-_CANVAS_HD_-_1945__THE_SAVAGE_PEACE.ts","Documentaires/Oorlogen/20170124_2110_-_CANVAS_HD_-_1945__THE_SAVAGE_PEACE.ts.ap","Documentaires/Oorlogen/20170124_2110_-_CANVAS_HD_-_1945__THE_SAVAGE_PEACE.ts.cuts","Documentaires/Oorlogen/20170124_2110_-_CANVAS_HD_-_1945__THE_SAVAGE_PEACE.ts.meta","Documentaires/Oorlogen/20170124_2110_-_CANVAS_HD_-_1945__THE_SAVAGE_PEACE.ts.sc"]
[2021-12-19 17:13:10.328598306] process [4217] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch-check=%(objectname) %(objecttype) %(objectsize)","--buffer"]
[2021-12-19 17:13:10.328861515] process [4218] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch=%(objectname) %(objecttype) %(objectsize)","--buffer"]
[2021-12-19 17:13:10.329676569] process [4219] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch=%(objectname) %(objecttype) %(objectsize)","--buffer"]
get Documentaires/Oorlogen/20170124_2110_-_CANVAS_HD_-_1945__THE_SAVAGE_PEACE.ts.meta [2021-12-19 17:13:10.341285273] process [4221] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","git-annex"]
[2021-12-19 17:13:10.342469002] process [4221] done ExitSuccess
[2021-12-19 17:13:10.342787111] process [4222] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","--hash","refs/heads/git-annex"]
[2021-12-19 17:13:10.343851119] process [4222] done ExitSuccess
[2021-12-19 17:13:10.34416143] process [4223] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","log","refs/heads/git-annex..33675b567939705c283bd955c2c7de13e15ba28a","--pretty=%H","-n1"]
[2021-12-19 17:13:10.345389634] process [4223] done ExitSuccess
[2021-12-19 17:13:10.345949274] process [4224] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch"]
[2021-12-19 17:13:10.346209136] process [4225] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch-check=%(objectname) %(objecttype) %(objectsize)"]
(from garage...)
[2021-12-19 17:13:10.349782417] process [4226] read: ssh ["-o","BatchMode=true","-S",".git/annex/ssh/root@10.19.20.147","-o","ControlMaster=auto","-o","ControlPersist=yes","-n","-T","root@10.19.20.147","true"]
[2021-12-19 17:13:11.687712914] process [4226] done ExitSuccess
[2021-12-19 17:13:11.68875019] process [4231] chat: ssh ["root@10.19.20.147","-S",".git/annex/ssh/root@10.19.20.147","-o","ControlMaster=auto","-o","ControlPersist=yes","-T","git-annex-shell 'p2pstdio' '/opt/Sat' '--debug' 'b22315b4-77f0-4f46-aed3-fe376b350ab9' --uuid dd4c0b26-2331-4133-82bf-1888753c3268"]
[2021-12-19 17:13:12.407472022] [ssh connection Just 4231] [ThreadId 4] P2P < AUTH-SUCCESS dd4c0b26-2331-4133-82bf-1888753c3268
[2021-12-19 17:13:12.40779008] [ssh connection Just 4231] [ThreadId 4] P2P > VERSION 1
[2021-12-19 17:13:12.423477213] [ssh connection Just 4231] [ThreadId 4] P2P < VERSION 1
[2021-12-19 17:13:12.423831707] [ssh connection Just 4231] [ThreadId 4] P2P > GET 165 Documentaires/Oorlogen/20170124_2110_-_CANVAS_HD_-_1945__THE_SAVAGE_PEACE.ts.meta SHA256E-s164--41ece560a700c3a1f2c1a28f37399c6cf139dde68f8c97ff9f7e10ac1080c746.ts.meta
[2021-12-19 17:13:28.731615325] [ThreadId 4] P2P < GET 165 Documentaires/Oorlogen/20170124_2110_-_CANVAS_HD_-_1945__THE_SAVAGE_PEACE.ts.meta SHA256E-s164--41ece560a700c3a1f2c1a28f37399c6cf139dde68f8c97ff9f7e10ac1080c746.ts.meta
[2021-12-19 17:13:28.796144027] [ThreadId 4] P2P > DATA 0
[2021-12-19 17:13:28.796944343] [ThreadId 4] P2P > VALID
[2021-12-19 17:13:12.518895068] [ssh connection Just 4231] [ThreadId 4] P2P < DATA 0
[2021-12-19 17:13:12.521393464] [ssh connection Just 4231] [ThreadId 4] P2P < VALID
[2021-12-19 17:13:12.522008877] [ssh connection Just 4231] [ThreadId 4] P2P > FAILURE
transfer failed
[2021-12-19 17:13:12.524417003] process [4224] done ExitSuccess
[2021-12-19 17:13:12.525545594] process [4225] done ExitSuccess
[2021-12-19 17:13:12.526125714] [ssh connection Just 4231] [ThreadId 4] P2P > GET 165 Documentaires/Oorlogen/20170124_2110_-_CANVAS_HD_-_1945__THE_SAVAGE_PEACE.ts.meta SHA256E-s164--41ece560a700c3a1f2c1a28f37399c6cf139dde68f8c97ff9f7e10ac1080c746.ts.meta
[2021-12-19 17:13:28.807577749] [ThreadId 4] P2P < FAILURE
[2021-12-19 17:13:28.812468221] [ThreadId 4] P2P < GET 165 Documentaires/Oorlogen/20170124_2110_-_CANVAS_HD_-_1945__THE_SAVAGE_PEACE.ts.meta SHA256E-s164--41ece560a700c3a1f2c1a28f37399c6cf139dde68f8c97ff9f7e10ac1080c746.ts.meta
[2021-12-19 17:13:28.812746153] [ThreadId 4] P2P > DATA 0
[2021-12-19 17:13:28.812854143] [ThreadId 4] P2P > VALID
[2021-12-19 17:13:12.533471116] [ssh connection Just 4231] [ThreadId 4] P2P < DATA 0
[2021-12-19 17:13:12.534230317] [ssh connection Just 4231] [ThreadId 4] P2P < VALID
[2021-12-19 17:13:12.534505767] [ssh connection Just 4231] [ThreadId 4] P2P > FAILURE
transfer failed
Unable to access these remotes: garage
[2021-12-19 17:13:12.537799752] process [4235] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch"]
[2021-12-19 17:13:12.538818173] process [4236] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch-check=%(objectname) %(objecttype) %(objectsize)"]
[2021-12-19 17:13:28.820206504] [ThreadId 4] P2P < FAILURE
No other repository is known to contain the file.
failed
[2021-12-19 17:13:12.55009435] process [4235] done ExitSuccess
[2021-12-19 17:13:12.550952683] process [4236] done ExitSuccess
[2021-12-19 17:13:12.551390026] process [4219] done ExitSuccess
[2021-12-19 17:13:12.551709257] process [4218] done ExitSuccess
[2021-12-19 17:13:12.551975888] process [4217] done ExitSuccess
[2021-12-19 17:13:12.552194866] process [4216] done ExitSuccess
[2021-12-19 17:13:12.561312091] process [4238] read: ssh ["-O","stop","-S","root@10.19.20.147","-o","ControlMaster=auto","-o","ControlPersist=yes","localhost"]
[2021-12-19 17:13:12.569198309] process [4238] done ExitSuccess
git-annex: get: 1 failed
As you can see the offset in the p2p get command is 165, which is exactly the file size of what it's supposed to be getting:
-r-xr-xr-x 1 root root 165 Apr 29 2020 .git/annex/objects/5V/x8/SHA256E-s164--41ece560a700c3a1f2c1a28f37399c6cf139dde68f8c97ff9f7e10ac1080c746.ts.meta/SHA256E-s164--41ece560a700c3a1f2c1a28f37399c6cf139dde68f8c97ff9f7e10ac1080c746.ts.meta
What version of git-annex are you using? On what operating system?
On the old server:
$ git annex version
git-annex version: 8.20200908
build flags: Assistant Webapp Pairing Inotify DBus DesktopNotify TorrentParser MagicMime Feeds Testsuite S3 WebDAV
dependency versions: aws-0.22 bloomfilter-2.0.1.0 cryptonite-0.26 DAV-1.3.4 feed-1.3.0.1 ghc-8.8.4 http-client-0.6.4.1 persistent-sqlite-2.10.6.2 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.1.0
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL X*
remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs httpalso hook external
operating system: linux x86_64
supported repository versions: 8
upgrade supported from repository versions: 0 1 2 3 4 5 6 7
local repository version: 8
On the new server:
$ git annex version
git-annex version: 8.20210223
build flags: Assistant Webapp Pairing Inotify DBus DesktopNotify TorrentParser MagicMime Feeds Testsuite S3 WebDAV
dependency versions: aws-0.22 bloomfilter-2.0.1.0 cryptonite-0.26 DAV-1.3.4 feed-1.3.0.1 ghc-8.8.4 http-client-0.6.4.1 persistent-sqlite-2.10.6.2 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.1.0
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL X*
remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs httpalso borg hook external
operating system: linux x86_64
supported repository versions: 8
upgrade supported from repository versions: 0 1 2 3 4 5 6 7
local repository version: 8
Both servers are debian.
Please provide any additional information below.
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log
# End of transcript or log.
Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
I've been using git annex for years now, some periods more intensively than others. The only data I've lost I can only blame on my own stupidity, even after a terminal disk failure I managed to recover almost everything with a reinject. I'm still using it entirely cli though, no assistants or gui for me.
fixed (the related issue that I discovered in comment #1 specifically) --Joey
It looks rather like the full content of the file may already have been downloaded, in
.git/annex/tmp/which is why it "resumes" from the end of the file.But this does not necessarily look like a protocol problem. The protocol handles no more data being sent. The failure is on the receiving side, after it's received the data it indicates an overall failure.
One reason this could be happening is if the file in
.git/annex/tmp/is corrupted, so verifying the completed download fails.I tried two experiements. First, I populated
.git/annex/tmp/with a copy of the annexed object file from the other repository. Thengit-annex getsucceeded, and looked this this:Next, I did the same, but corrupted the tmp file. With that I reproduced your problem exactly.
So, I suggest you try deleting files in
.git/annex/tmp/and get them again.I think that git-annex should probably try to recover from this by, when verification of a downloaded file fails, deleting the tmp file. But that will need some more thought.
Hi Joey,
Thank you for having a look at this, but we can put this one under "my own stupidity". The files that give these errors are apparently only status files which get rewritten every time you play a recording, so they continuously get rewritten, get "git annex add"ed with a cron job, but don't always get pushed before they get overwritten, so yeah, it's not unexpected that some of these may get lost at some point. Still no real data lost though, but errors like these worry me enough to file a bug report.
regards, Tim
Err, unless you have
annex.thinset, once yougit-annex adda file, it should not be possible for it be overwritten in a way that loses the annexed copy of it.Ah, I see.
I am going to leave this bug open, since the corrupted tmp file problem I found is a real problem, though not actually your problem.
Current thinking on deleting corrupted tmp files: If a download succeeds, and verification then fails, the whole file content has been downloaded, and is corrupt. So it would be ok to always delete it then, as far as p2p transfers goes.
For other remotes, the same is often true. The only exceptions are like rsync and bittorrent, which can recover from corruption on retry. But, I don't think either rsync or bittorrent will usually write corrupt data to a file anyway. They would catch over-the-wire corruption with rolling checksums etc. So, it seems like a verification should never fail after a successful rsync or bittorrent download. Unless the disk corrupted the data in the meantime. Which is an unlikely situation, and not one that it's really necessary for git-annex to recover from with optimal efficiency.
... Oh interesting.. It already is supposed to do that, in getViaTmpFromDisk. It seems, what is happening is the transfer fails when all the file content is present, and so it never gets to the point of verifying it, let alone deleting it.
This seems to be a reversion caused by incremental verification. In P2P.Annex it does incremental verification, and so it notices that the temp file is corrupted. So, the transfer fails, and the temp file is left behind.
So I suppose the fix is that, whenever doing incremental verification, also delete the temp file when it fails.