Please describe the problem.
when running git annex status
i get 'alien' content in the list of repositories (some timestamps, something looking like git status messages, and a long list of (recovery from race)
messages. the actual content seems fine and the list of repositories in the assistant looks ok, but the CLI status messages show this unusual content (full dump below).
What steps will reproduce the problem?
not sure how i would go about reproducing this: i only noticed this weird output just now and i'm pretty sure it was fine 24h ago - in the meanwhile the single non-ordinary operation was deleting via the assistant's webapp an SSH remote, rsync+encryption, then creating it again as bare repo without encryption.
after the assistant finished removing all the content from the previous annex directory, it stalled reporting deletion in progress (the annex directory's parent on the remote server was not writeable for the user used for the ssh connection - perhaps that was the cause). even after chowning the parent dir and restarting the assistant on my local workstation, the assistant would still show the remote as pending deletion.
after a while i could add the remote again, setting it to be a git bare repo, and content was copied over.
from the assistant's webapp, i still see the correct list of remotes, but the output of git annex status
is as below.
.git/config
also looks ok, listing some core settings, some annex settings and my three real remotes for this annex.
the part of logs relevant to the above change (deletion of remote, creation of new one + content sync) is unfortunately gone from the annex logs (including the rotated ones in .git/annex/daemon.log*
).
What version of git-annex are you using? On what operating system?
4.20130709:amd64 on Debian testing (linux 3.9.8-1)
Please provide any additional information below.
$ git annex status
supported backends: SHA256E SHA1E SHA512E SHA224E SHA384E SHA256 SHA1 SHA512 SHA224 SHA384 WORM URL
supported remote types: git S3 bup directory rsync web webdav glacier hook
repository mode: direct
trusted repositories: 7
1359161911.638435s -- 1
1359161913.173474s -- 1
1359161913.288686s -- 1
1359332402.294867s -- 1
1359332406.839623s -- 1
1359332407.454511s -- 1
1359332411.932021s -- 1
semitrusted repositories: 36
00000000-0000-0000-0000-000000000001 -- web
00511808-8d7c-11e2-a252-43e07a9aaa3c -- client
12527a3a-a21d-4cc7-a7ea-074e8fd0260f -- annex_on_rpi00 (standard)
12a7b66c-e681-4dcc-bf75-b995512e5e76 -- client
1359159154.357141s -- 1 2ce1be52-6745-11e2-83e5-e7d111798afd
1359159155.817851s -- 1 2ce1be52-6745-11e2-83e5-e7d111798afd
1359159155.931382s -- 1 e08336a0-6742-11e2-8a53-17c5ab00507f
1359161255.198519s -- 1 2ce1be52-6745-11e2-83e5-e7d111798afd
1359161256.72633s -- 1 2ce1be52-6745-11e2-83e5-e7d111798afd
1359161256.952372s -- 1 e08336a0-6742-11e2-8a53-17c5ab00507f
1359332437.573825s -- 1 2e71af50-674c-11e2-b542-6397738d1182
1359332440.782882s -- 0 2e71af50-674c-11e2-b542-6397738d1182
1359332441.153755s -- 1 42c358c4-6741-11e2-ba3f-8bdecec02757
1359332448.091981s -- 0 2ce1be52-6745-11e2-83e5-e7d111798afd
1359332507.354892s -- 1 2e71af50-674c-11e2-b542-6397738d1182
1359332512.553832s -- 1 2e71af50-674c-11e2-b542-6397738d1182
1359332512.666779s -- 1 42c358c4-6741-11e2-ba3f-8bdecec02757
1359332517.773994s -- 1 42c358c4-6741-11e2-ba3f-8bdecec02757
1359332523.112182s -- 0 2ce1be52-6745-11e2-83e5-e7d111798afd
1374091795.70517s -- 1 2e71af50-674c-11e2-b542-6397738d1182
1374507271.515001s -- 0 8d243fde-0c4c-4a19-8bce-ac1beb48eef0
1374507272.066961s -- 0 8d243fde-0c4c-4a19-8bce-ac1beb48eef0
1a76c11b-b359-4606-be1d-cc3b17dc684e -- client
2ce1be52-6745-11e2-83e5-e7d111798afd -- git.services__data_annex_ (transfer)
2e71af50-674c-11e2-b542-6397738d1182 -- here (client)
42c358c4-6741-11e2-ba3f-8bdecec02757 -- x11miel.local_data_annex (em@10.10.10.100:~/data/annex)
4eb75def-169c-45f8-b9b0-a0aa52017bd6 -- client
92aa538c-8d7b-11e2-9ae1-3fd2531928c1 -- client
author -- nnnn nnnn <em@a.il> 1374509958 +0100
committer -- nnnn nnnn <em@a.il> 1374509958 +0100
d83e4134-8d7a-11e2-8822-7f2218f2cc42 -- client
e08336a0-6742-11e2-8a53-17c5ab00507f -- backup
f7969bd0-d5dc-11e2-b362-cbb323f72415 -- transfer
merging -- 10.10.10.10__mnt_annex_00_annex/git-annex into git-annex (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race) (recovery from race)
parent -- fcde6fd44ff2ad2aaca895bca1b32fb398d76545
tree -- b6d4b49d9b67b296b0a0d9a800b96f88d480f923
untrusted repositories: 2
1359332417.999925s -- 0
1374507270.94356s -- 0
transfers in progress: none
available local disk space: 6 gigabytes (+1 megabyte reserved)
local annex keys: 1984
local annex size: 6 gigabytes
known annex keys: 1989
known annex size: 6 gigabytes
bloom filter size: 16 mebibytes (0.3% full)
backend usage:
SHA256E: 3973
$
i have now updated the git-annex package to the latest 4.20130723:amd64 still on Debian testing (linux 3.9.8-1) - i see the same issue (while files keep synchronising fine across the various connected nodes).
i have noticed that the same output appears when i issue
git annex status
on the other nodes (an Ubuntu 13.04 with kernel 3.9.7 amd64 and a Debian stable with 3.10.1-1 amd64, all with latest Debian 4.20130723:amd64 package).Please paste the output of:
git show git-annex:uuid.log
The file seems to have a lot of garbage in it. The "merging -- 10.10.10.10__mnt_annex_00_annex/git-annex into git-annex (recovery from race)" in particular is only ever passed to git as the commit message. It should not be possible for it to appear in a file unless data has somehow become corrupted.
So, I think you should also run
git fsck
. You may also find that it helps to shut down any running git-annex, and move.git/annex/index
away to a backup location. A corrupted index file would be kind of be the best scenario since git-annex can automatically rebuild it from the git repository..thanks Joey.
output from
git show git-annex:uuid.log
is below.git fsck
shows me 4201 lines ofdangling blob <sha>
.git annex fsck
instead checksums all the files and only reports a single error about no known copies of an individual file. i guess this can be relied upon to confirm that the actual data is fine?i have also tried shutting down any running git-annex and moving the index file away - it was rebuilt when i restarted the assistant, but i still get the same output from
git annex status
.i should probably work around this by just copying the annex folder's contents to a new folder and create a new annex there (perhaps in indirect mode rather than direct as it is now, as this is mostly a document archive with no further changes to files except some occasional renaming) and create my remotes from scratch, but now i'm quite curious about trying to track down what happened and to understand if the annex is reliably storing and syncing content even though the status message contains alien data.
(output from
git show git-annex:uuid.log
follows)The 4201 dangling blobs is a little bit strange, although it could certainly happen in some normal scenarios. Overall, your repository seems to be ok except for this weird data in the one file. I do not anticipate the extra garbage causing any problems at all.
To track this down, we need to find the commit that added the garbage. One way, assuming you're using indirect mode, is to
git checkout git-annex; git blame uuid.log
andgit show
the commit that added the garbage. If you're using direct mode, you should firstgit clone
the repository and do that in the clone.thanks; the current
uuid.log
file carries content from 8 distinct commits :would the whole diff for each commit be useful, or just the part related to uuid.log?
all these commits were done by git-annex on my laptop; the earliest of these commits come seconds after two commits done on another node - a raspberry pi running armhf Raspbian, although i now see that
git-annex
andgit-annex-shell
on this rPi are constantly segfaulting (though i can't understand the reason from the strace output): perhaps a segfault while dealing with a git operation on the rPi started the metadata corruption somehow?btw - if this could be useful to track down a possible bug in git-annex (although only involving some metadata, not the actual data), i am happy to dig further to provide all the information needed to understand what might have happened, but if it's not something worth investigating further in the bigger scheme of things, from my point of view i'm ok with restarting a fresh annex with content moved from this annex, and pristine metadata![:)](../../smileys/smile.png)
I think that the whole
git show $sha
for each commit will provide useful context.I want to understand what happened here, because there are multiple weird things going on, and it's important to root cause this sort of bug.
What version of git-annex is on the rPi and how was it installed? Is there a simple operation you can do that segfaults every time?
thanks Joey. i'm sending the output of
git show $sha
separately as it contains some non-public data; according to apt/dpkg logs, the rPi was actually using git-annex:armhf 4.20130521 from Debian testing at the time these commits were done. i did in fact install 4.20130709 and then 4.20130723 a couple of days after the issue showed up, but only yesterday i realised that both versions were segfaulting all the time (just invokinggit-annex
would segfault); 4.20130521 works reliably, instead (i have just downgraded to test), so my previous question about an interrupted operation possibly causing the metadata corruption is not valid as the git-annex executable in use at the time of the issue was not segfaulting (at least not under every circumstance as the more recent versions in my setup - but i'll dig into this separately - probably some library issue or so).the only change i can't really trace in terms of timing is when i deleted the remote i had originally set up on my rPi (using encrypted rsync) via the assistant's web interface on my laptop, and set it up again as plain unencrypted ssh remote (bare repo). i'm pretty sure this operation was done the day the commits in the
git blame
foruuid.log
were done, but i'm not sure about the time and don't have logs going back to that moment.i could disconnect all the remotes from my laptop and redo the setup on the rPi to see if i can reproduce the issue, but i would need some pointers on how to revert my main annex on my laptop to the state it was in before i first started using the rPi as remote: can i just
git reset --hard <some_sha>
while on thegit-annex
branch? the annex is currently in direct mode (and as been so for a few months now).otherwise, i could easily copy all the data (or a subset, to start with) to a clean folder, set up a fresh annex there and pair the rPi again - if the corruption was due to some issue on the rPi it may not depend on the exact status the annex was at when the metadata corruption happened (although my full setup includes other remotes and i see some back-and-forth of merging going on throughout the history of the annex around the time the metadata issue started, so it may be hard to reproduce the exact same context).
dpkg -i
on the packages downloaded directly from packages.d.o.Here's an excerpt of the first commit where uuid.log got garbage into it. It seems that some other log files got garbage put in them before this point, and it will be interesting to see if every time new garbage gets in it's doing a "(recovery from race)" commit. If so, that would point pretty strongly at that code as a culprit.
(Of course, it's intriguing that the race recovery commit itself seemed to encounter a race and this repeated dozens of times. An exceptionally unlikely thing to happen if everything is behaving correctly.)
Looks a lot like a whole git commit object was union merged into the file, doesn't it?