I guess this is a mismatch between the version of git that git-annex was built with and the version it's using. In particular, git check-attr behavior varies between git older than 1.7.7 and newer, and git-annex picks the version at build time.
So, I think this is a broken Ubuntu PPA, and if that's the current one, the maintainer of the PPA needs to be contacted to update the git-annex to match the git version, or depend on an appropriate git version.
I will look into PPA. I hope it won't be this slow:) How long would it take for git annex to add/modify/sync files (~100) taking up 100GB or so? Days? Hours? Minutes?
Adding the files will take as long as it takes to read and hash
the conents of those files from disk. If that's too slow, --backend=WORM
will bypass the hashing, so it will take seconds.
Time required to sync files depends on the bandwidth to wherever it's
syncing with, obviously.
I have seen a similar sounding version on Centos 6.5. I had git version 2.6.4 and installed git annex using:
wget http://download.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
rpm -ivh epel-release-6-8.noarch.rpm
yum install git-annex
git annex would hang when running 'git annex add'
When I changed the git version to be 1.7.1 this problem went away. So it is something to do with the combination of client git version and git annex version.
This was fixed in git-annex back in January 2015. It now checks the git
version at runtime, not assuming it will be used with as recent a version
of git as was available at compile time.
So if you're still seeing this problem, upgrading git-annex or git is the only
solution.
The setup was working quite fine for a while and now the problem just appeared. Actually, right before I noticed the problem I did git-annex move [...] -J10 from another machine to this repo but it didn't seem to work correctly, so I pressed CTRL-C..
I cannot reboot the server, since I don't have root access.
"The server" where my repo is stored are actually 4 servers for load balancing that share a single file system.
I just found out that on one of the servers the were still some git-annex-shell processes running. Probably the move command with -J10 logged in on different servers; maybe this confused git-annex.
Now, I logged-in on all four servers and killed any remaining processes. Still the problem persists.
git-annex info --debug
[2017-05-17 16:29:07.197350961] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","git-annex"]
[2017-05-17 16:29:07.216435634] process done ExitSuccess
[2017-05-17 16:29:07.216539092] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","--hash","refs/heads/git-annex"]
[2017-05-17 16:29:07.2214318] process done ExitSuccess
[2017-05-17 16:29:07.2220868] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","log","refs/heads/git-annex..76cf27fbf3697af8843ddb82f1fffde35aaba0b2","--pretty=%H","-n1"]
[2017-05-17 16:29:07.23077964] process done ExitSuccess
What's interesting is that these killing actions apparently have changed the hash number on which the hang happens (76cf..). What I also did was git-annex sync at another host, this actually seemed to work. Maybe this changed the hash.. I'm not entirely sure..
I'm not sure what exactly has changed, but it's working again!
Until now, I was using git-annex version: 6.20170321-gf3dee9d65 from the standalone-amd64.tar.gz. In the meantime I convinced the administrator to install git-annex system wide. She installed git-annex version: 5.20140221 from the packet-system.
Now, the repository works again with both, git-annex 5 and 6.
However, I cannot exactly tell if anything else has changed in the meantime, since I am not the administrator of the system.
Probably the same problem reported here: https://github.com/datalad/datalad/issues/12
I guess this is a mismatch between the version of git that git-annex was built with and the version it's using. In particular, git check-attr behavior varies between git older than 1.7.7 and newer, and git-annex picks the version at build time.
So, I think this is a broken Ubuntu PPA, and if that's the current one, the maintainer of the PPA needs to be contacted to update the git-annex to match the git version, or depend on an appropriate git version.
Adding the files will take as long as it takes to read and hash the conents of those files from disk. If that's too slow,
--backend=WORMwill bypass the hashing, so it will take seconds.Time required to sync files depends on the bandwidth to wherever it's syncing with, obviously.
I have seen a similar sounding version on Centos 6.5. I had git version 2.6.4 and installed git annex using: wget http://download.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm rpm -ivh epel-release-6-8.noarch.rpm yum install git-annex
git annex would hang when running 'git annex add'
When I changed the git version to be 1.7.1 this problem went away. So it is something to do with the combination of client git version and git annex version.
This was fixed in git-annex back in January 2015. It now checks the git version at runtime, not assuming it will be used with as recent a version of git as was available at compile time.
So if you're still seeing this problem, upgrading git-annex or git is the only solution.
Hi,
it seems my git-annex repository is also affected by the "hanging" bug.
Any operation of git-annex just hangs and doesn't do anything. Problem occurs with: sync, whereis, fsck (and with everything I tested).
Then it just hangs..
Also here, hangs..
I "installed" git-annex from the pre-compiled binaries (i.e., copied it somewhere and put a simlink to git-annex and git-annex-shell into $HOME/bin).
Versions are:
The setup was working quite fine for a while and now the problem just appeared. Actually, right before I noticed the problem I did git-annex move [...] -J10 from another machine to this repo but it didn't seem to work correctly, so I pressed CTRL-C..
I cannot reboot the server, since I don't have root access.
Actually the situation is even more complex.
"The server" where my repo is stored are actually 4 servers for load balancing that share a single file system.
I just found out that on one of the servers the were still some git-annex-shell processes running. Probably the move command with -J10 logged in on different servers; maybe this confused git-annex.
Now, I logged-in on all four servers and killed any remaining processes. Still the problem persists.
What's interesting is that these killing actions apparently have changed the hash number on which the hang happens (76cf..). What I also did was git-annex sync at another host, this actually seemed to work. Maybe this changed the hash.. I'm not entirely sure..
So there is some kind of networked filesystem involved? What kind?
Do you have annex.pidlock enabled in the repository's
git config?I checked the file system, it's nfs.
git config --get annex.pidlockgives nothing, so I guess it's not enabled. Do I have to enable it, if the repo is on an nfs mount?Hi,
I'm not sure what exactly has changed, but it's working again!
Until now, I was using git-annex version: 6.20170321-gf3dee9d65 from the standalone-amd64.tar.gz. In the meantime I convinced the administrator to install git-annex system wide. She installed git-annex version: 5.20140221 from the packet-system.
Now, the repository works again with both, git-annex 5 and 6.
However, I cannot exactly tell if anything else has changed in the meantime, since I am not the administrator of the system.
Thanks for your help, anyway.