Please describe the problem.
A git-annex get fails with createDirectory: does not exist (No such file or directory) on MacOSX.
What steps will reproduce the problem?
I can trigger the condition as part of a CI run of the DataLad test suite. Here is an example run that shows the failure: https://ci.appveyor.com/project/mih/datalad/build/job/k5u263619e6erk8t However, the exact conditions required to trigger the issue are not yet known (c.f. https://github.com/datalad/datalad/issues/5291).
A protocol of an exploration of this issue with debug output is here: https://github.com/datalad/datalad/issues/5301#issuecomment-757467813
In this end I can trigger the error with a mkdir performed manually in the shell using the path reported by git-annex (e.g. ../../../../../../../var/folders/5s/g225f6nd6jl4g8tshbh1ltk40000gn/T/datalad_temp_tree_test_basic_scenariodi3ady04/.git/annex/). But a mkdir is successful using a "normalized variant of the path
pointing to the same physical directory (e.g. /var/folders/5s/g225f6nd6jl4g8tshbh1ltk40000gn/T/datalad_temp_tree_test_basic_scenariodi3ady04/.git/annex).
I do not understand enough of this platform to understand what is happening, but it seems that some of these folders actually live in /private/var, despite being accessible via /var, but I do not see how mkdir would error on a relative path and succeed on an absolute one.
What version of git-annex are you using? On what operating system?
8.20201129 on darwin/19.6.0 10.15.7/x86_64
More system details are here https://ci.appveyor.com/project/mih/datalad/build/job/k5u263619e6erk8t#L422
Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
git-annex is critical infrastructure for me. There is no day without it. Thx much!
So there's an OSX bug here, and git-annex has been made to use an absolute path to a remote when it has one, which can be used to work around the OSX bug. done --Joey
/tmp, which maps to some place under/private).Is the relative path actually valid? There may be some case involving symlinks in the parent directories where the relative path could be wrong. This part seems to suggest there are symlinks involved:
Of course, it's a bit surprising that a relative path is used when the path is into an entirely different top-level directory. But without understanding the problem I don't know if switching to using an absolute path in that case would only happen to fix this case of the problem and not the general case, whatever that is.
Also it seems common for directories under /var/folders/xx/yyyyy/ to be mode 700, so it could somehow involve permissions.
As far as I can tell, it is.
I cannot come up with an explanation, for the consistent behavior of
ls, but notmkdirfor a relative path vs. an absolute path.I can't see how git-annex can work around this, unless someone finds an explanation for the OS's behavior.
I think you've clearly shown it's not a git-annex bug, since other mkdir() calls behave the same way. And that seems to point squarely at an OS bug.
(Unless .git/annex was itself a dangling symlink, then ls would be able to list it, but mkdir would fail like that.)
I feel it is OS (OSX bug), does not reproduce on newer 11.1 (appveyor has 10.15.7), asked on SO anyways with minimalistic example. I believe it is the
tmp -> private/tmpsomehow throws it off.But I also wonder why git-annex uses relative path for get whenever remote is specified via an absolute one?
IIRC it might have been done to minimize the path length so we have a better chance to not hit limits. But in this case, if anything - it makes it only longer.
PS initially posted with another "wisdom" that relative path was off but only then spotted that it was
/varvs/private/varAgreed, what you show in the stackoverflow post can only be an OSX bug.
Using a relative path for the current repository does avoid using long absolute paths, and also has a minor benefit of letting the repository be moved while git-annex is running and it still accessing the right files.
Neither benefit applies to the paths to remotes. I think the only reason relative paths are being used for them is that the same code is used to operate on them as on the local repository.
I've made some changes that will make it use absolute paths when the remote has an absolute path.