Recent comments posted to this site:

Just noticed another thing: If that failure happens, git-annex exits zero. Therefore it's hard to detect for us. We have just an empty json as return value as if the repo was clean. It would be nice to have either a non-zero exit or something within the json to indicate that the command failed.

ben

Comment by benjamin.poldrack Wed Feb 22 16:48:04 2017

Yes, it saves half the work and eases to deal with this one.

ben

Comment by benjamin.poldrack Tue Feb 21 06:20:34 2017

Would it suffice for datalad to have git-annex status --ignore-submodules pass the option on to git status?

Comment by joey Mon Feb 20 20:13:06 2017
"""
Comment by joey Mon Feb 20 19:15:18 2017

Yes, that's what I assumed. Therefore this bug was linked in the datalad issue regarding --ignore-submodules.

May be we are the only ones, who are actually effected, since we heavily use submodules. If it's too much of an effort, I will find a way around it, I guess. Either way: Thank you for having a look at it. ben

Comment by benjamin.poldrack Mon Feb 20 19:06:58 2017

Probably wget is just failing to download the url sometimes. Eg, git annex addurl http://localhost/dne fails with the same not useful output.

wget is run with -q, which is the only way to turn off all its informational messages, but unfortunately that also turns off display of HTTP error messages.

Using -nv instead of -q would display HTTP errors, but also 1 extra line of output once the download is complete. I suppose that's worth the trade-off.

Comment by joey Mon Feb 20 18:50:18 2017

When you used embedcreds=yes, it committed the creds to the git-annex branch of the git repository. For embedcreds=no to do anything useful, it would need to remove that data from the git repository history.

Removing data from a git repository tends to involve rewriting commits and forced pushes to all remotes, it's not a simple process and not ameanable to automation. It will be much easier, and more secure, to go into S3 and generate new credentials, and revoke access to the old ones.

What git annex enableremote with embedcreds=no does do is prevent any new creds from being embedded into the repository. Otherwise, git annex enableremote will update the embedded creds with whatever new ones are set in the environment when you run it.

Comment by joey Mon Feb 20 18:38:16 2017

Error is:

fatal: This operation must be run in a work tree
fatal: 'git status --porcelain' failed in submodule sub

git status fails the same way, and it seems that the only way to make it work would be to pass --ignore-submodules to it.

But I suppose then, it would need to replicate git status's submodule traversal.

I'm not too keen on adding complicated stuff involving submodules to direct mode. My goal with direct mode is to eliminate the need for it.

Comment by joey Mon Feb 20 18:30:54 2017

Joey,

the file names seem quite ordinary:

cv/submissions/BCA_Submission/small_images/drawblocks_2015_IMG_1719.jpg
cv/submissions/BCA_Submission/small_images/drawblocks_2015_IMG_3848.jpg
cv/submissions/BCA_Submission/small_images/macropavilion_2016_IMG_0391.jpg
cv/submissions/BCA_Submission/small_images/sequencing_2016_DSC5048.jpg
cv/submissions/BCA_Submission/small_images/sequencing_2016_IMG_8231.jpg

Possibly this is related to an issue I had with v6 and annex.largefiles about 12-months ago. I had done git config annex.largefiles 'largerthan=100kb and not (include=*.c or include=*.h)'. But I believe this resulted in git-annex thinking locked files with no content present should be added to git instead of annex? After doing a git-annex add . I now had a bunch of files whose content was lost. Or perhaps I did a sync with a v5 repo, or perhaps I did a sync with a repo with a different large files settings, I can't remember. Anyhow, I managed to lose links to the file content and git-annex get or fsck couldn't retrieve them. I never filed a bug report because I was never able to reproduce the issue on a clean repo.

Anyhow, perhaps spelunking into the git log would give some answers?

Andrew

Comment by andrew Mon Feb 20 18:11:03 2017

Error message is:

git-annex: unexpected object type "comm"

What it's actually choking on is the "commit" object for the submodule, in git-ls-tree output. Doesn't matter if the submodule uses adjusted branches or not.

The parser for ls-tree output is buggy; it's expecting only "blob" and "tree", so pulls out a fixed width 4 characters: "comm"

Also, the adjusted branch code needs to be made to skip over CommitObjects, once the parser is fixed to generate them.

Comment by joey Mon Feb 20 17:13:54 2017