Please describe the problem.
I'm getting fatal: unable to normalize object directory while issuing git annex copy from an annex in my laptop to a vanilla remote annex located in a USB disk.
As far as I can tell there are no hardware problems with the disk or the laptop.
``` $ git annex copy --auto --to usbdrive . copy vlc-record-2016-07-17-15h16m55s-hr-fernsehen-Die fantastische Reise der Vögel (3_6).ts (to usbdrive...) SHA256E-s1256869488--28edfd2c4eb08e6782c806a85db7bc10595c16df6f64d3d46ebc2802b3ba5038.ts 1,256,869,488 100% 52.96MB/s 0:00:22 (xfr#1, to-chk=0/1) (checksum...) fatal: unable to normalize object directory: /home/jr/Videos/ts/../../../../../run/media/jr/TOSHIBA/jr/Videos.git/objects fatal: unable to normalize object directory: /home/jr/Videos/ts/../../../../../run/media/jr/TOSHIBA/jr/Videos.git/objects fatal: unable to normalize object directory: /home/jr/Videos/ts/../../../../../run/media/jr/TOSHIBA/jr/Videos.git/objects fatal: unable to normalize object directory: /home/jr/Videos/ts/../../../../../run/media/jr/TOSHIBA/jr/Videos.git/objects fatal: unable to normalize object directory: /home/jr/Videos/ts/../../../../../run/media/jr/TOSHIBA/jr/Videos.git/objects fatal: unable to normalize object directory: /home/jr/Videos/ts/../../../../../run/media/jr/TOSHIBA/jr/Videos.git/objects fatal: unable to normalize object directory: /home/jr/Videos/ts/../../../../../run/media/jr/TOSHIBA/jr/Videos.git/objects fatal: unable to normalize object directory: /home/jr/Videos/ts/../../../../../run/media/jr/TOSHIBA/jr/Videos.git/objects fatal: unable to normalize object directory: /home/jr/Videos/ts/../../../../../run/media/jr/TOSHIBA/jr/Videos.git/objects fatal: unable to normalize object directory: /home/jr/Videos/ts/../../../../../run/media/jr/TOSHIBA/jr/Videos.git/objects fatal: unable to normalize object directory: /home/jr/Videos/ts/../../../../../run/media/jr/TOSHIBA/jr/Videos.git/objects
fd:24: hGetLine: end of file
SHA256E-s1256869488--28edfd2c4eb08e6782c806a85db7bc10595c16df6f64d3d46ebc2802b3ba5038.ts 1,256,869,488 100% 56.08MB/s 0:00:21 (xfr#1, to-chk=0/1) (checksum...) fd:23: hFlush: resource vanished (Broken pipe)
fd:23: hClose: resource vanished (Broken pipe) ok (recording state in git...) ```
But note that the command did not abort. After copying, fsck reports no error:
$ git annex fsck --from=usbdrive 'vlc-record-2016-07-17-15h16m55s-hr-fernsehen-Die fantastische Reise der Vögel (3_6).ts'
fsck vlc-record-2016-07-17-15h16m55s-hr-fernsehen-Die fantastische Reise der Vögel (3_6).ts (checksum...) ok
(recording state in git...)
I get the same errors with drop, which fails:
``` $ git annex drop --from usbdrive 'vlc-record-2016-07-17-15h16m55s-hr-fernsehen-Die fantastische Reise der Vögel (3_6).ts' drop usbdrive vlc-record-2016-07-17-15h16m55s-hr-fernsehen-Die fantastische Reise der Vögel (3_6).ts fatal: unable to normalize object directory: /home/jr/Videos/ts/../../../../../run/media/jr/TOSHIBA/jr/Videos.git/objects fatal: unable to normalize object directory: /home/jr/Videos/ts/../../../../../run/media/jr/TOSHIBA/jr/Videos.git/objects fatal: unable to normalize object directory: /home/jr/Videos/ts/../../../../../run/media/jr/TOSHIBA/jr/Videos.git/objects fatal: unable to normalize object directory: /home/jr/Videos/ts/../../../../../run/media/jr/TOSHIBA/jr/Videos.git/objects fatal: unable to normalize object directory: /home/jr/Videos/ts/../../../../../run/media/jr/TOSHIBA/jr/Videos.git/objects fatal: unable to normalize object directory: /home/jr/Videos/ts/../../../../../run/media/jr/TOSHIBA/jr/Videos.git/objects fatal: unable to normalize object directory: /home/jr/Videos/ts/../../../../../run/media/jr/TOSHIBA/jr/Videos.git/objects fatal: unable to normalize object directory: /home/jr/Videos/ts/../../../../../run/media/jr/TOSHIBA/jr/Videos.git/objects fatal: unable to normalize object directory: /home/jr/Videos/ts/../../../../../run/media/jr/TOSHIBA/jr/Videos.git/objects fatal: unable to normalize object directory: /home/jr/Videos/ts/../../../../../run/media/jr/TOSHIBA/jr/Videos.git/objects fatal: unable to normalize object directory: /home/jr/Videos/ts/../../../../../run/media/jr/TOSHIBA/jr/Videos.git/objects
git-annex: fd:27: hGetLine: end of file failed git-annex: drop: 1 failed ```
But the same drop works if executed again:
$ git annex drop --from usbdrive 'vlc-record-2016-07-17-15h16m55s-hr-fernsehen-Die fantastische Reise der Vögel (3_6).ts'
drop usbdrive vlc-record-2016-07-17-15h16m55s-hr-fernsehen-Die fantastische Reise der Vögel (3_6).ts ok
(recording state in git...)
What version of git-annex are you using? On what operating system?
git-annex version: 6.20161231-gc8eeb17da
Linux XXX 4.8.13-1-ARCH #1 SMP PREEMPT Fri Dec 9 07:24:34 CET 2016 x86_64 GNU/Linux
Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
Yes! I use git-annex without a problem since years. This is the first time I have really no idea what's going on.
I found the problem: I was cd'ing into the local repository via a symbolic link, instead of using the "canonical" path. You can mark the bug as solved or delete it. Sorry for the noise.
"fatal: unable to normalize object directory" is an error message that git emits, not git-annex.
I wonder if git-annex's use of --work-tree and --git-dir with relative paths confuses git in some way?
It would be good to explain how to reproduce this problem. I tried a few things involving symlinks in the path I
cded to, but was not able to cause the problem.$ cd /; sudo ln -s /mnt/hd/ .; cd /hd/repo; git annex copy --to=someremote ./some-fileexhibits the problem. Thanks!Thanks for the reproducion recipe, @rfourquet. That worked for me.
The bug submitter closed this bug, but I'm reopening it since this seems a legitimate bug report and we know how to reproduce it too.
With somerepo in ~/tmp/repo, and pwd in /hd/repo, which is a symlink from /tmp/hd/repo, this happens:
Here it's making git operate on the remote git repo. The relative paths it uses to it seem legitimate, in that "ls ../.." shows the content of "/tmp" and "ls ../../.." shows the root directory.
This is a very confusing situation, and different ways of getting the current directory give different results in this situation. In particular
$PWDcontains "/hd/repo" while getcwd(3) will return "/tmp/hd/repo". Paths need to be relative to "/tmp/hd/repo" as that's the actual cwd.It seems that git is looking at
$PWDto get "/hd/repo" and normalizing the relative path via that. So there's no possible relative path that git will accept in this situation. This kind of seems like buggy behavior on git's part. I've posted about it to the git mailing list. https://marc.info/?l=git&m=151008314122623&w=2git-annex could avoid the problem by not using relative paths of course, but there are reasons including shorter path length (generally) for its use of relative paths.
git-annex could unset PWD before running git, which forces git to fall back to using getcwd(3). I've verified that does avoid the problem. Going to see if this just gets fixed in git before I put in such ugly and kind of expensive workarounds. (Expensive because, to unset PWD from the environment, git-annex would have to copy it.)
The git bug is fixed in git's
pubranch, or as Peff says, at least "papered over", and I've tested the fix with git-annex and it seems to avoid the problem. So as long as 5eb0e541132208a9027cc090943276fa52f29c71 gets into a git release, this bug can be closed without any changes to git-annex.