Please describe the problem.
subject
What steps will reproduce the problem?
repo from http://datasets.datalad.org/workshops/nih-2017/ds000114/derivatives/freesurfer/
What version of git-annex are you using? On what operating system?
6.20170525+gitge1cf095ae-1~ndall+1
Please provide any additional information below.
[student3@helix freesurfer]$ 'git' '-c' 'receive.autogc=0' '-c' 'gc.auto=0' 'annex' 'get' '--debug' '--json' '--json-progress' 'sub-01/label/lh.aparc.a2009s.annot'
[2017-07-30 09:33:42.479678681] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","ls-files","--cached","-z","--","sub-01/label/lh.aparc.a2009s.annot"]
[2017-07-30 09:33:42.501443872] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","git-annex"]
[2017-07-30 09:33:42.523673251] process done ExitSuccess
[2017-07-30 09:33:42.523765181] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","--hash","refs/heads/git-annex"]
[2017-07-30 09:33:42.544559473] process done ExitSuccess
[2017-07-30 09:33:42.553475534] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch"]
[2017-07-30 09:33:42.554641292] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch-check=%(objectname) %(objecttype) %(objectsize)"]
[2017-07-30 09:33:42.591306199] call: curl ["-sS","-f","-L","-C","-","-#","-o","/scratch/nih-2017/ds000114/derivatives/freesurfer/.git/annex/tmp/MD5E-s831572--7437ad482af573329af54386a15007fe","https://dl.dropbox.com/s/85yms1fl56pq7io/lh.aparc.a2009s.annot?dl=0","--user-agent","git-annex/6.20170525+gitge1cf095ae-1~ndall+1"]
curl: (77) error setting certificate verify locations:
CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
[2017-07-30 09:33:42.693467186] process done ExitFailure 77
{"command":"get","wanted":[{"here":false,"uuid":"00000000-0000-0000-0000-000000000001","description":"web"},{"here":false,"uuid":"5e47b3f3-f09c-4969-8885-920a49ff8a45","description":"yoh@smaug:/mnt/btrfs/datasets/datalad/crawl/workshops/nih-workshop-2017/ds000114/derivatives/freesurfer"}],"note":"Try making some of these repositories available:\n\t00000000-0000-0000-0000-000000000001 -- web\n \t5e47b3f3-f09c-4969-8885-920a49ff8a45 -- yoh@smaug:/mnt/btrfs/datasets/datalad/crawl/workshops/nih-workshop-2017/ds000114/derivatives/freesurfer\n","skipped":[],"success":false,"key":"MD5E-s831572--7437ad482af573329af54386a15007fe","file":"sub-01/label/lh.aparc.a2009s.annot"}
git-annex: get: 1 failed
[student3@helix freesurfer]$ 'git' '-c' 'receive.autogc=0' '-c' 'gc.auto=0' 'annex' 'get' '--json' '--json-progress' 'sub-01/label/lh.aparc.a2009s.annot'
curl: (77) error setting certificate verify locations:
CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
{"command":"get","wanted":[{"here":false,"uuid":"00000000-0000-0000-0000-000000000001","description":"web"},{"here":false,"uuid":"5e47b3f3-f09c-4969-8885-920a49ff8a45","description":"yoh@smaug:/mnt/btrfs/datasets/datalad/crawl/workshops/nih-workshop-2017/ds000114/derivatives/freesurfer"}],"note":"Try making some of these repositories available:\n\t00000000-0000-0000-0000-000000000001 -- web\n \t5e47b3f3-f09c-4969-8885-920a49ff8a45 -- yoh@smaug:/mnt/btrfs/datasets/datalad/crawl/workshops/nih-workshop-2017/ds000114/derivatives/freesurfer\n","skipped":[],"success":false,"key":"MD5E-s831572--7437ad482af573329af54386a15007fe","file":"sub-01/label/lh.aparc.a2009s.annot"}
git-annex: get: 1 failed
while works fine without:
[student3@helix freesurfer]$ 'git' '-c' 'receive.autogc=0' '-c' 'gc.auto=0' 'annex' 'get' 'sub-01/label/lh.aparc.a2009s.annot'
get sub-01/label/lh.aparc.a2009s.annot (from web...)
/scratch/nih-2017/ds000114/ 100%[===========================================>] 812.08K --.-KB/s in 0.08s
2017-07-30 09:34:02 URL:https://dl.dropboxusercontent.com/s/85yms1fl56pq7io/lh.aparc.a2009s.annot?dl=0 [831572/831572] -> "/scratch/nih-2017/ds000114/derivatives/freesurfer/.git/annex/tmp/MD5E-s831572--7437ad482af573329af54386a15007fe" [1]
(checksum...)
(recording state in git...)
git-annex no longer uses either curl or wget by default, and always uses curl when configured to do so, so this kind of surprising behavior will no longer occur done --Joey
Using --json will have the same effect as --quiet in that commands have to be made to not output to stdout in that mode. In the case of downloading an url, git-annex usually uses wget, however when curl is available in path, it uses it in quiet mode, because curl displays error messages to stdout, whereas wget never displays error messages at all when run in quiet mode.
So, by using --json, you're changing the command that git-annex uses to download files from wget to curl.
But, the git-annex standalone build bundles both wget and curl, so I'd expect that any certificates problem would affect both equally. But I suppose wget may be better able to deal with the certs on this system than curl, for whatever reason.
Bundling certs with git-annex seems wrong on multiple counts; it's moving a large security boundary inside git-annex; it's lots of work; there may be local policies on allowed certs that this would disallow.
Suggestion: Remove curl from the standalone build you're using, so it will use the system-wide curl.
There is no guarantee that system would have curl installed, e.g. I believe neither of debian docker images would have curl. For neurodebian package I sure thing could just depend on curl. But I am afraid those who might be using standalone git-annex in minimalistic environments would get surprised
FWIW -- removing curl from the bundled git-annex indeed helped