Greetings all!
I have only been using git-annex for a few days, but I am loving it so far.
I have run into one problem:
At the top of a directory tree with some very large (GB+) blobs, I issue this command:
find . -size +5M -exec git-annex add --debug {} \; 2>&1 | tee -a ./annex-add.log
When the command completes, I see a number of these types of errors in the log:
[2015-10-05 21:14:13.502615] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch"]
[2015-10-05 21:14:13.521586] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch"]
[2015-10-05 21:14:13.533095] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","diff","--name-only","--diff-filter=T","-z","--","./application/mozilla/firefox/linux/firefox-39.0/media/firefox-39.0.tar.bz2"]
[2015-10-05 21:14:13.545739] feed: xargs ["-0","git","--git-dir=.git","--work-tree=.","--literal-pathspecs","add","--"]
[2015-10-05 21:14:13.562727] process done ExitSuccess
[2015-10-05 21:14:13.564103] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","hash-object","-w","--stdin-paths","--no-filters"]
[2015-10-05 21:14:13.56616] feed: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","update-index","-z","--index-info"]
[2015-10-05 21:14:13.58032] process done ExitSuccess
fatal: Out of memory, getdelim failed
[2015-10-05 21:14:13.582109] process done ExitFailure 128
ok
(recording state in git...)
git-annex: user error (git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","hash-object","-w","--stdin-paths","--no-filters"] exited 128)
I've googled around and searched the forums/bug reports here without much success. I have tried setting the queuesize to smaller values, but that does not seem to help.
I am using git-annex version: 5.20150930-g40fdbe9 on RHEL 6.
I am happy to help debug further...
Thanks,
JC
P.S. - Is it normal that some links in the .git dir are created without user write access?
This message about "getdelim" comes from git, not git-annex. It seems git hash-object is what's running out of memory, and probably when it's processing its input from stdin.
That's pretty strange. git-annex just passes some filenames to git hash-object. Unless the system is for some reason very low on memory, it shouldn't choke on such a small input as a filename.
This seems likely to be a git bug, so what version of git? Perhaps try a newer one..
(Re: P.S.: It's normal for files in .git/annex/objects to not have write access.)
Thank for the response Joey...
Here's my git version:
I've doubled the amount of RAM on that VM to 8GB - I'll see what happens when I try another run.
Had another report of this, and there NFS seemed to be involved in the circumstances of the crash.
NFS was a red herring.
We've tracked this down to the linux standalone tarball not including locale-archive, so it uses the system one, which may have a different format and so breaks glibc.
I'm committing a fix that adds locale-archive to the linux standalone tarball.
https://github.com/datalad/datalad/issues/1678 git annex standalone is quite recent so those manpage workarounds should be in place
The locale generation done by the standalone tarball needs write access to its directory. If datalad is using the git-annex-standalone.deb, non-root user's won't be able to write to it, and it will still use the system locales.
Installed a stretch based singularity container on a Centos 6.9 with 2.6.32-696.6.3.el6.x86_64 kernel and within it, I reliably get the message
also note that message might come not the last one in the output (initial run for the command was different):
ha -- and then later call succeeded! (damn... right before I was trying to debug it)
but comes back anyways ;)
pretty sure in that
[[! format sh """ yhalchen@discovery:/mnt/scratch/yoh/datalad$ git pull --ff-only origin master fatal: Out of memory, getdelim failed error: git://github.com/datalad/datalad did not send all necessary objects
yhalchen@discovery:/mnt/scratch/yoh/datalad$ git pull --ff-only origin master fatal: Out of memory, getdelim failed error: git://github.com/datalad/datalad did not send all necessary objects
yhalchen@discovery:/mnt/scratch/yoh/datalad$ git pull --ff-only origin master From git://github.com/datalad/datalad * branch master -> FETCH_HEAD fatal: Out of memory, getdelim failed
"""]]
Yes, I think that the git-annex-standalone.deb is potentially broken by mismatches between the system locales and the bundled version of glibc.
The git-annex standalone tarball was designed to be unpacked by the user who's going to use it; having root install it is not using it as it was designed. If you're going to have root install something, why not have them install a docker container or some other container format designed to be installed by root, that can completely isolate the application inside it?
(As to the getdelim error specifically, here's a link to the thread where that bug in git was tracked down and patched https://public-inbox.org/git/20170810202942.s232yr5stjxmmz4t@hopa.kiewit.dartmouth.edu/. I'm still not clear on how locale issues might contribute to the getdelim error. It seems like something before the getdelim call is failing in this situation for whatever reason, and that might be the actual root cause.)