Please describe the problem.
git annex status reports 160 semitrusted repositories. Four of them are the ones I created (only via webapp, I think I never edited any git-annex config file). One is 00000000-0000-0000-0000-000000000001 -- web and although I do not know what it is, it is not something new. The remaining 155 appeared spontaneously after several Gb of data (mostly many small files) were added to an Annex (in an 'archive' directory) operated in direct mode by the assistant.
What steps will reproduce the problem?
Add several Gb of files was enough to trigger this problem, but I did not try to reproduce it. It happened the day I installed the 4.20131106~bpo70+1 version.
What version of git-annex are you using? On what operating system?
4.20131106~bpo70+1 on debian squeeze (7.2), with git 1.8.4.rc3.
Please provide any additional information below.
May be related or not: at some point the webapp displayed two warning boxes. One of them held a message that I did not wirte down and proposed to "Restart the thread". This apparently worked since the box disappeared. The other warning box indicated "NetWatcherFallback crashed: unknown response from git cat-file" and proposed to restart the thread. Trying to "restart the thread" via the provided button just did not trigger any response of the webapp which seemed dead at that point.
In spite of the git annex status shown below, the webapp still shows only the expected four repositories.
Output of git annex status (hostname and xmpp account name were edited away):
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log
repository mode: direct
trusted repositories:
0
semitrusted repositories: 160
00000000-0000-0000-0000-000000000001 -- web
0ab193eb-0c76-4559-a93c-2e30ed8630a8 -- someMachineIown_datadir (archive)
1384784127.91222s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
1384784164.437824s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
1384784176.944372s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
1384784179.254498s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
1384785147.558938s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
1384785147.717223s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
1384785159.041203s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
1384785159.199504s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
1384785185.79485s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
1384785187.318128s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
1384785215.236504s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
1384785215.389096s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
1384785313.539843s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
1384785313.701305s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
1384785315.596206s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
1384785344.184461s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
1384785348.192805s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
1384785402.70316s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
1384785406.524044s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
1384785446.074236s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
1384873605.313126s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
1384873697.029999s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
1384873761.687234s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
1384873774.608376s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
1384926279.456728s -- 0 391b0557-dc68-4e40-b6d0-da3033588753
1384926368.736s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
1384926454.99433s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
1384926494.152645s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
1384926504.438232s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
1384934790.89717s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
1384934848.757067s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
1384934899.087168s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
1384934908.238587s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
1384948772.14552s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
1384948805.441196s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
1384948813.397132s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
1384948921.45481s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
1384948924.855852s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
1384949073.988946s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
1384949082.298976s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
1384949399.608138s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
1384949581.12213s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
1384949583.9923s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
1384949700.22807s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
1384949765.484768s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
1384955202.85962s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
1384955230.953995s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
1384955402.534938s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
1384955457.1885s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
1384955524.603709s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
1384955611.891061s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
1384955677.84592s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
1384955689.293082s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
1384955894.057476s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
1384955910.723021s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
1384955914.732132s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
1384955968.717875s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
1384955969.634658s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
1384956004.284925s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
1384956029.567195s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
1384956188.628995s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
1384956379.844701s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
1384956381.613833s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
1384956387.923418s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
1384956395.418701s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
1384956408.792928s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
1384956504.019733s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
1384956519.578085s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
1384956524.419783s -- 1 391b0557-dc68-4e40-b6d0-da3033588753
1384965891.562742s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
1384965891.815119s -- 0 391b0557-dc68-4e40-b6d0-da3033588753
1384965903.355602s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
1384965905.276128s -- 0 391b0557-dc68-4e40-b6d0-da3033588753
1384965978.806653s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
1384965979.393089s -- 0 391b0557-dc68-4e40-b6d0-da3033588753
1384966097.495566s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
1384966097.704474s -- 0 391b0557-dc68-4e40-b6d0-da3033588753
1384966154.97658s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
1384966156.967406s -- 0 391b0557-dc68-4e40-b6d0-da3033588753
1384966233.310488s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
1384966233.522324s -- 0 391b0557-dc68-4e40-b6d0-da3033588753
1384966241.284523s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
1384966241.475381s -- 0 391b0557-dc68-4e40-b6d0-da3033588753
1384966301.688497s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
1384966303.427685s -- 0 391b0557-dc68-4e40-b6d0-da3033588753
1384966392.875983s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
1384966393.38718s -- 0 391b0557-dc68-4e40-b6d0-da3033588753
1384966404.708568s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
1384966406.441164s -- 0 391b0557-dc68-4e40-b6d0-da3033588753
1384966553.557387s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
1384966555.752786s -- 0 391b0557-dc68-4e40-b6d0-da3033588753
1384966653.725847s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
1384966654.23288s -- 0 391b0557-dc68-4e40-b6d0-da3033588753
1384966695.201885s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
1384966695.689398s -- 0 391b0557-dc68-4e40-b6d0-da3033588753
1384966784.556877s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
1384966786.574886s -- 0 391b0557-dc68-4e40-b6d0-da3033588753
1384966791.446852s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
1384966793.218318s -- 0 391b0557-dc68-4e40-b6d0-da3033588753
1384966884.335685s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
1384966886.147083s -- 0 391b0557-dc68-4e40-b6d0-da3033588753
1384967054.857465s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
1384967055.158871s -- 0 391b0557-dc68-4e40-b6d0-da3033588753
1384967190.980027s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
1384967193.176584s -- 0 391b0557-dc68-4e40-b6d0-da3033588753
1384967328.93796s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
1384967330.428095s -- 0 391b0557-dc68-4e40-b6d0-da3033588753
1384967526.127311s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
1384967526.588491s -- 0 391b0557-dc68-4e40-b6d0-da3033588753
1384967627.132549s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
1384967627.685201s -- 0 391b0557-dc68-4e40-b6d0-da3033588753
1384967686.283694s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
1384967686.728086s -- 0 391b0557-dc68-4e40-b6d0-da3033588753
1384967768.270887s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
1384967768.58402s -- 0 391b0557-dc68-4e40-b6d0-da3033588753
1384967769.245615s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
1384967771.122238s -- 0 391b0557-dc68-4e40-b6d0-da3033588753
1384967813.8197s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
1384967814.168477s -- 0 391b0557-dc68-4e40-b6d0-da3033588753
1384967915.243469s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
1384967917.020051s -- 0 391b0557-dc68-4e40-b6d0-da3033588753
1384968031.757775s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
1384968032.190452s -- 0 391b0557-dc68-4e40-b6d0-da3033588753
1384968035.733635s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
1384968036.03299s -- 0 391b0557-dc68-4e40-b6d0-da3033588753
1384968144.555556s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
1384968144.714535s -- 0 391b0557-dc68-4e40-b6d0-da3033588753
1384968150.090148s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
1384968150.820567s -- 0 391b0557-dc68-4e40-b6d0-da3033588753
1384968304.393177s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
1384968304.613624s -- 0 391b0557-dc68-4e40-b6d0-da3033588753
1384968604.499519s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
1384968604.813256s -- 0 391b0557-dc68-4e40-b6d0-da3033588753
1384968702.566939s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
1384968704.427767s -- 0 391b0557-dc68-4e40-b6d0-da3033588753
1384968725.375289s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
1384968725.939271s -- 0 391b0557-dc68-4e40-b6d0-da3033588753
1384968798.402904s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
1384968798.659754s -- 0 391b0557-dc68-4e40-b6d0-da3033588753
1384969055.285004s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
1384969055.715448s -- 0 391b0557-dc68-4e40-b6d0-da3033588753
1384969159.885115s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
1384969162.382266s -- 0 391b0557-dc68-4e40-b6d0-da3033588753
1384969184.633052s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
1384969185.413769s -- 0 391b0557-dc68-4e40-b6d0-da3033588753
1384969374.791849s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
1384969377.497842s -- 0 391b0557-dc68-4e40-b6d0-da3033588753
1384969475.469111s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
1384969489.697737s -- 0 391b0557-dc68-4e40-b6d0-da3033588753
1384969492.087023s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
1384969492.58214s -- 0 391b0557-dc68-4e40-b6d0-da3033588753
1384969784.725195s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
1384969786.49773s -- 0 391b0557-dc68-4e40-b6d0-da3033588753
1384969814.984624s -- 1 0ab193eb-0c76-4559-a93c-2e30ed8630a8
1384969815.397676s -- 0 391b0557-dc68-4e40-b6d0-da3033588753
391b0557-dc68-4e40-b6d0-da3033588753 -- here (client)
668ef9d8-68c6-484e-89e5-06634d590a11 -- rsync.net_datadir_annex (transfer)
b0d3c000-0ac9-4a05-aef4-47f826d5c759 -- user.name (client)
# End of transcript or log.
git show git-annex:uuid.logrun in the repository.Here it is (user name edited):
Any chance I could get a copy of this git repository? (Excluding the .git/annex part)
Feel free to email me id@joeyh.name to arrange a secure transfer.
Otherwise, I need to see what the git-annex:uuid.log file looked like before this happened to it. The corruption may have occurred progressively in several commits, or all at once.
You might also still have some logs in
.git/annex/daemon.log*, and sending those might help, assuming the strange messages you mentioned are logged in there.This is pretty weird thing to be in the uuid.log. The "1" makes me think this might be a scrambled version of what's normally stored in the trust.log:
This seems to be the same bug as non-repos in repositories list (+ other weird output) from git annex status
One client is a thinkpad 121e, the other one a hp Z420 workstation. The archive is a basic Dell desktop. All of them run wheezy with git-annex and git from official backports. The transfer repository is at rsync.net.
The daemon.log file was empty at the time I noticed the situation. Now git-annex has been scanning this repository for some time and the log has information again but it looks a bit too long to be posted here. I will email about the repository copy and the log.
git annex getsays it gets data in an amount which seems correct but the link to the data remains broken. Fortunately I can still access the data on the other client. Some of the data in the archive is still accessible in both clients. The part that is still accessible was put in the annex a few days ago, only the last directory which was added and seems to have triggered the problem is "broken" on one client. It is possible (not sure, sorry) that this directory was first added in the annex, and then moved to the archive shortly after, before the first addition had propagated to the other client.OK, sorry but it will be difficult to provide you with a copy of this repository. I hope I can still help in spite of being only a very novice git user. I checked that the git-annex:uuid.log file at the first commit was identical to the one above. The only difference is that in this first commit file there is a mention at the bottom of who created the file (me@somedomain.com) with the mention "created repository".
So it is very possible that these nonexisting remotes are present since day 1. It is possible that I did not notice it earlier (I am not sure I had ever run git annex status in this repository since I am looking for a dropbox-like experience). What triggered my curiosity was the difficulty I had to get some data. At that point it might be that there are two independent problems. One with these nonexisting repositories, and another one with the fact that I cannot get some data in this client.
Would it be useful to you to have the full history (52 revisions) of the git-annex:uuid.log file?
Yes, it would be helpful to have the full history of the file.
You might try running
git annex fsckon the file in the client that it says it has gotten but that you cannot access.git annex fsckand for all files the command returnedok. Nevertheless, it was still not possible to get the data. Now after a few days it seems I can get all the data I tried from both clients. Maybe because the repository contains many files (about 1.5e6) the daily sanity check spawns git processes which do not seem to terminate even when the delay has expired but other than that the repository does not seem to be in a bad state, the only weirdness being the 155 nonrepositories in the status.It's normal for the git-annex assistant to keep a couple of long-running git cat-file and git check-attr processes running at all times. So I'd not worry about that.
I received the emailed tarball of uuid.log versions. (message-id CAL5AKQq9i4QwReWHZkKOm1Ggon66FqFkjbM832PP1drgiR3x5g@mail.gmail.com)
Can you show the command you used to dump those files? Because the contents of them is really quite weird.
Take file 0, which I'm guessing might be the first version of the uuid.log. It starts off with what I'd expect to see in a uuid.log, but then has a section of what looks like trust.log data, followed by what is certianly supposed to be remote.log data, followed by what looks like
git show 966ec7aaf408dffe0d6e6ce04aebc15dc5b26f5b, a commit which apparently was made by "Firstname.Lastname@somedomain.com" (unless you anonymized that) and added a filetestto the git repository.It gets worse, some of the versions of the uuid.log file appear to be megabytes in size, which is just insane, and these contain more such apparent
git showof git commits adding files to the repository.The general pattern of uuid.log data +trust.log data + remote.log data + git show of a commit seems to repeat in each version of the uuid.log file.
The files I sent you were collected with:
for i in $(seq 0 52); do git show git-annex:uuid.log @{$i} > ~/tmp/uuid.logs/$i; doneOk, that explains the weird results, because I get the same crazy data dumped when I try eg
git show git-annex:uuid.log @{0}in a repository that is completely ok.The @{0} causes git show to show some commit. Not the commit that changed uuid.log however. So it first shows the uuid.log (always the most recent version), and then some random commit.
This will work:
I followed the procedure you suggested, please see email for how to transfer it to you. There are several points I would like to note:
git annex statusdoes not contain all the weird things that appear in the original repository (although it contains other stuff, see 3)git annex statusstill contains the 155 nonrepositories, plus some other stuff (see 3)git annex statuson client1 in ~/datadir/Annex shows the repositories of ~/private/Annex in addition to the 155 nonrepositories and its own repositoriesIn the end the situation is now the following:
I am sorry that the situation is that messy. Is there a way to separate these repositories that have somehow become entangled?