Please describe the problem.
Ref: Original issue and finding report against datalad.
What version of git-annex are you using? On what operating system?
Windows 8.20200815-g335aae266 (I see now that I might benefit from update, but I found no related issue so likely it is something new)
Please provide any additional information below.
{920}[Level 5] CommandError: '"git" "annex" "add" "-c" "annex.dotfiles=true" "-c" "annex.retry=3" "--json" "--json-error-messages" "--debug" "--" "file.dat"' failed with exitcode 1 under C:\Users\DataLad\AppData\Local\Temp\datalad_temp_tree_q4wxn05d [out: '{"command":"add","success":false,"error-messages":[" file.dat failed to link to annex"],"file":"file.dat"}'] [err: '[2020-10-15 11:57:07.4657537] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.dotfiles=true","-c","annex.retry=3","show-ref","git-annex"]
| [2020-10-15 11:57:07.4813787] process done ExitSuccess
| [2020-10-15 11:57:07.4813787] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.dotfiles=true","-c","annex.retry=3","show-ref","--hash","refs/heads/git-annex"]
...
| [2020-10-15 11:57:07.6376287] call: cp ["--reflink=auto","-a","file.dat",".git\\annex\\objects\\6f0\\fbd\\SHA256E-s7--ed7002b439e9ac845f22357d822bac1444730fbdb6016d3ec9432297b9ec9f73.dat\\SHA256E-s7--ed7002b439e9ac845f22357d822bac1444730fbdb6016d3ec9432297b9ec9f73.dat"]
| "cp": unrecognized option `--reflink=auto'
| Try `"cp" --help' for more information.
| [2020-10-15 11:57:07.6532537] process done ExitFailure 1
C:\tmp>cp --help | grep ref
-d same as --no-dereference --preserve=link
-L, --dereference always follow symbolic links
-P, --no-dereference never follow symbolic links
C:\tmp>cp --version
cp (GNU coreutils) 5.97
Copyright (C) 2006 Free Software Foundation, Inc.
This is free software. You may redistribute copies of it under the terms of
the GNU General Public License <http://www.gnu.org/licenses/gpl.html>.
There is NO WARRANTY, to the extent permitted by law.
Written by Torbjorn Granlund, David MacKenzie, and Jim Meyering.
if I parsed NEWS.gz correctly, that option was added in * Noteworthy changes in release 8.0 (2009-10-06) [beta]
. So I wonder if git-annex
could sense for version of cp
or presence of --reflink
option and not use it if cp
is too old?
Meanwhile I will see if there is some sensible way to get more uptodate coreutils (or cp
) at least within conda env
a mental note... building git-annex in a windows vm ATM, and saw
so either the check is wrong or there are multiple
cp
around (may be some came withstack
?) ... I found total over 10 different cp.exe on drive (didn't check yet how many in PATH). But I just wonder, if check is done at build time, butcp
is not distributed along, the check should be done at run time instead.git.exe
is at/c/Program Files (x86)/Git/cmd/git.exe
(pasting from mingw'ed bash) whilecp
is under/c/Program Files (x86)/Git/usr/bin/cp.exe
so not exactly the same directory, but I guess consistent for all Windows'es.I'm not going to make git-annex probe everything at run time. The user can find more ways to break their environment than I can probe for, and sheer code complexity, and runtime complexity, and debug complexity, and run time would all be affected.
The way git-annex is installed into git for windows's path is supposed to both make "git annex" work and also put it in the same directory that git for windows installs curl and cp. IIRC on windows a program automatically looks in its directory along with the path, so that makes them available to git-annex. There are probably configurations that make some other path be used in preference, or perhaps something changed in git for windows's locations of these things, although the path you show sounds like the same one I'd expect git-annex.exe to be in. I do know that used to work.
--reflink seems unlikely to do anything useful on windows even if the option is supported, so it could just be defaulted to not supported on windows. -a/-p/--preserve-timestamps are more important and also probed at build time.
I installed git for windows in a fresh windows 10 VM. Then I installed git-annex from the git-annex for windows installer. Then I ran git-annex add, which ran cp --reflink successfully.
But you didn't say how you installed git-annex on windows, did you? I guess, since you're building it yourself, you put it into path somewhere else. So it seems to me, it's the up to you to make its path include the same cp you built it against.
Or just build the windows installer too, and install your git-annex build using that.
It isn't exactly "how git-annex installed" but rather, what else is installed besides git-annex. Recommended way to install datalad on OSX and Windows is via anaconda or miniconda from conda-forge channel. See So it places itself into
PATH
ahead of the rest, and it is the one which comes ATM with outdatedcp
. It seems thatcp
might come also withgit
package there, and I have initiated (but didn't progress) effort to build newer coreutils for Windows for conda-forge. (there is also some other m2sys channel with somewhat newer coreutils build, but mixing channels that much is begging for more of other issues). So, eventually, I hope that there would be updatedcp
available by default withdatalad
installation, but the point is that unless git-annex uses the specificcp
it knows about, it could be some othercp
in its way.yeap, sounds good to me ATM.
If I read NEWS correctly, it was added in 4.1.10, so old enough (was I even born then?)
git-annex.exe
is in exactly the sameC:\Program Files (x86)\Git\usr\bin
ascp
etc -- so specifying specific path of wheregit-annex.exe
installed (I bet the process would know which dir it is coming from) tocp
and all those tools for a windows build could potentially make those installations more robust overall.If whatever is putting itself at the front of PATH, I think it would also risk breaking any part of git that expects to run another part of it by path. Including any git commands that use the bundled cp or other unix commands.
Perhaps the problem is that you're mixing two different package systems? If there's going to be a datalad package, there should also be a git-annex package, and then it can be built to use the right cp options for its distibution.
Working in the same environment as in the other issue. It does come with more recent
cp
(coreutils) 8.25 BUT that one blows with (sorry -- this is debug output of datalad, not direct output from git/annex invocation):so it echoes your "-a/-p/--preserve-timestamps are more important and also probed at build time." and is yet another gotcha of using different build and run environments. I only wonder how is that our github windows workflow environment supports
cp
with preserving permissions while conda's doesn'tSince this bug was opened, some uses of cp have been converted to git-annex doing the copy itself without cp, first probing for CoW support.
I do not think that the places mentioned in this bug report where cp fails are ones that have been converted yet. The most likely way to fix this bug would be to convert them. But the need to remember what directory combinations it's probed for CoW support in and so avoid probing for CoW support every time complicates that conversion. See related: ?strong>
But I don't think git-annex will ever stop relying on programs bundled with it (when not packaged with a package manager that supports deps). It's much easier to reimplement cp than gpg or ssh. So these version skews are always a potential problem, when people are doing things that put other programs ahead of the ones bundled with git-annex in PATH. So the benefit of avoiding external cp entirely just doesn't seem compelling.
I've disabled using --reflink=auto on windows. Going to close this with that.