I am getting errors when I try to do sync --content. I have a git-annex repo on my laptop and I sync all content to an S3 compatible remote (Dreamhost DreamObjects). I ran the following command in my attempt to better understand the issue:
andrew@bumblebee ~/notes$ git-annex --verbose --debug sync --content cloud &> /tmp/sync_log_2016-11-7-7pm.log
For reference the repos preferred content settings are:
andrew@bumblebee ~/notes$ git-annex wanted here
standard
andrew@bumblebee ~/notes$ git-annex group here
manual
andrew@bumblebee ~/notes$ git-annex wanted cloud
standard
andrew@bumblebee ~/notes$ git-annex group cloud
backup
andrew@bumblebee ~/notes$
Looking at the log file I see errors like this, repeated dozens of times:
fatal: '../Users/andrew/notes/cv/submissions/BCA_Submission/small_images/macropavilion_2016_IMG_0297.jpg' is outside repository
[2016-11-07 19:34:37.542383] process done ExitFailure 128
[2016-11-07 19:34:37.542473] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","check-attr","-z","--stdin","annex.backend","annex.numcopies","annex.largefiles","--"]
fatal: '../Users/andrew/notes/cv/submissions/BCA_Submission/small_images/macropavilion_2016_IMG_0297.jpg' is outside repository
[2016-11-07 19:34:37.551241] process done ExitFailure 128
[2016-11-07 19:34:37.551382] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","check-attr","-z","--stdin","annex.backend","annex.numcopies","annex.largefiles","--"]
The relative path ../Users is curious. /Users/andrew is the absolute path of my home directory (I am on OS-X 10.11.6); /Users/andrew/notes is the absolute path of my git-annex repo.
the above logs are followed by:
git-annex: git check-attr EOF: user error
failed
git-annex: fd:24: hFlush: resource vanished (Broken pipe)
failed
hFlush lines appears hundreds of times, then change to:
git-annex: fd:24: commitBuffer: resource vanished (Broken pipe)
failed
commitBuffer lines also apear hundreds of times.
Then the logs move onto successful transfers, and similar errors repeat with different file names.
I have tried git annex fsck on the local repo, and also tried git annex fsck --fast --from=cloud (although I didn't let it finish) and didn't see any errors. Thoughts?
Thanks, Andrew
I am getting the same errors when syncing to my USB drive. So it does not appear to have anything to do with syncing to the cloud.
Andrew
Any thoughts? I am unsure how to investigate where this problem is. I assume these files are in my git repo or git-annex objects but I can't seem to find them using any search commands.
Thanks,
Andrew
It would be helpful if you said what version of git-annex you are using.
And, is your git-annex repository using the new experimental v6 format? One user reported a similar error message with a v6 git-annex repository. See ?assistant crashes in TransferScanner
Or might your repository be using direct mode?
So, please paste in
git annex versionandgit annex infooutput.It kind of looks like it's having difficulty determining where the top of the git repository is, or constructing a relative path to the git repository.
Are there any symlinks in the path to /Users/andrew/notes ? Eg, is /Users a symlink, or /Users/andrew a symlink, or //Users/andrew/notes itself symlinked to elsewhere?
Does only
git annex sync --contentfail? What if you run, eggit annex copy --auto --to cloudandgit annex get --auto --from cloud, does that fail similarly, or does it succeed?You say it's only failing for some files. Do the filenames that it's failing on contain any non-ascii characters?
Yes, only
git annex sync --contentseems to fail. I am using v6 with a mix of unlocked and locked files. I did not know about the --auto flags for copy/get.git annex copy --auto --to cloudworks finegit annex get --auto --from cloudworks fineAre there any symlinks in the path to /Users/andrew/notes ? Eg, is /Users a symlink, or /Users/andrew a symlink, or //Users/andrew/notes itself symlinked to elsewhere?
No
You say it's only failing for some files. Do the filenames that it's failing on contain any non-ascii characters?
They seem normal.
So, please paste in git annex version and git annex info output.
Any thoughts Joey? Reminder I have a v6 repo, that spits out errors
git-annex: fd:24: hFlush: resource vanished (Broken pipe)when using sync --content.I am not sure why
git annex sync --content cloudshould spew errors, andgit annex copy --auto --to cloudruns without issue. Anything I should do on my end to investigate this issue further?Thanks,
Andrew
OK. I ended up having to delete all the files from git, then re-add them zipped up.
I re-read the post you mentioned, this helped me resolve my issue: assistant crashes in TransferScanner (from johannes)
Johannes dropped the file, then re-added, that fixed his issue. So I tried that first.
I did the following:
Joey,
the file names seem quite ordinary:
Possibly this is related to an issue I had with v6 and
annex.largefilesabout 12-months ago. I had donegit config annex.largefiles 'largerthan=100kb and not (include=*.c or include=*.h)'. But I believe this resulted in git-annex thinking locked files with no content present should be added to git instead of annex? After doing agit-annex add .I now had a bunch of files whose content was lost. Or perhaps I did a sync with a v5 repo, or perhaps I did a sync with a repo with a different large files settings, I can't remember. Anyhow, I managed to lose links to the file content andgit-annex getor fsck couldn't retrieve them. I never filed a bug report because I was never able to reproduce the issue on a clean repo.Anyhow, perhaps spelunking into the git log would give some answers?
Andrew
Hmm, maybe.
It's very difficult to say without enough of a transcript to have any idea at what point in the
git annex syncit was failing. If it failed in the "commit" part, then yes, it could be git commit running the smudge/clean filter and the worktree path problem affecting it.