Please describe the problem.
I am running testremote on a windows CI system to test a special remote implementation for dataverse.org. I run into this error:
git-annex: MoveFileEx "C:\\DLTMP\\ran2133" Just ".git\\annex\\objects\\f76\\373\\SHA256E-s1048576--813fea02438e9569e6222f802958fcd89bee742d06ffe9aabe27fd940ef01196.this-is-a-test-key\\SHA256E-s1048576--813fea02438e9569e6222f802958fcd89bee742d06ffe9aabe27fd940ef01196.this-is-a-test-key": does not exist (The system cannot find the path specified.)
I suspect this could be a path-length issue (the system reports a max length of 285, and the relative path given above is already 230 chars.
I thought to run git annex testremote --backend=MD5E instead, to shorten the key length, but this options is not honored (enough), the error showing a SHA256 key remains the same.
testremote man page says "Also the git-annex-common-options(1) can be used." and --backend is explicitly listed in the help output, hence I assumed this should work.
What steps will reproduce the problem?
It happens when running the https://github.com/datalad/datalad-dataverse tests on a windows appveyor worker. Running on a crippled FS is not enough to trigger the initial testremote error, it only happens on windows proper. However, I assume that --backend not having the effect that I assumed it should have, is not platform specific.
Here is a demo test log: https://ci.appveyor.com/project/mih/datalad-dataverse/builds/44079592/job/b38woai0ekmq7bn5#L856
The corresponding datalad issue is https://github.com/datalad/datalad-dataverse/issues/127
What version of git-annex are you using? On what operating system?
CI used
- annex: 8.20211117-gc3af94eff
- git: 2.37.0.windows.1
Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
All the time! Sorry to mostly show up when there is an issue!
Actually testremote will not accept --backend in current master, since that is no longer a global option and is accepted only by commands that can actually use it.
testremote cannot support an arbitrary backend here, because it needs to generate a test key that cannot possibly be used for real data. The only backend that has a way implemented to do that is SHA256. It would not, for example, be possible to make the WORM backend support that, since every possible WORM key could be used by real data.
It would be possible to add support for --backend=MD5 and have it reject other backends. But this does not strike me as solving the real problem.
Also, in ?tests fail on windows: retrieveKeyFile resume I ran into this same problem, when
git-annex testwas ran, and worked around it by disabling that part of the test suite on windows. If this is fixed, it would be worth re-enabling that, although it may have also been failing for other reasons on windows.ghc's IO manager tries to support Windows long paths by normalizing to an UNC-style path in many system calls. However, when git-annex calls rename, on windows that ends up in Win32's moveFileEx (via unix-compat), and that does not do UNC-style normalization. And given the description of the Win32 package, I think it's intended to pass data directly through to the API without anything fancy.
System.Directory.renamePath could be used instead of Win32. While it still uses Win32 moveFileEx, it first does an UNC-style normalization. Filed an issue: https://github.com/jacobstanley/unix-compat/issues/56
Rather than waiting for that to be fixed, I've made git-annex use System.Directory.renamePath instead itself. But I don't know if it will be enough to make testremote work, or if it will fall over on a later operation on the same too-long path. getFileStatus/getSymbolicLinkStatus seem like the main things in unix-compat that would still be a problem.
Sorry for the long silence. Coming back to this issue I find the behavior changed, but not sufficiently to get the test suite to run in full on windows.
I ran
git annex testremote --faston Windowsmsys_nt-10.0-17763with git-annex 10.20230227-ga206cdddb4 and git 2.38.1.windows.1Show test output
hide
Right now, I cannot say whether this is pointing to a problem in my implementation or still to something in git-annex. However, the same implementation passes the test suite on linux.
Sidenote: I am not sure of you have access to a windows system for debugging. If this is needed or helpful, please let me know.
Thanks!
Seems like my renamePath fix did work, because looking back at the origianl failure log, it was failing to generate test keys, before it got to run the test cases at all.
The new failures seem likely to be due to getFileStatus/getSymbolicLinkStatus failing on the long filename on windows, as I suspected might happen in comment #3. I've updated the issue at https://github.com/jacobstanley/unix-compat/issues/56. And maybe that will get fixed, my understanding is that unix-compat has a new maintainer recently. But, git-annex does contain a convertToNativeNamespace function that it could use to work around the problem itself.
(I am able to run Windows in emulation, but it's sufficiently slow and disk hog that I generally am not in a position to do it easily and appreciate users who can save me the bother.)
In 54ad1b4cfb1c8302f1b862cb2699ab9351e3eb5b I fully worked around this class of problems with unix-compat.
I think it's reasonably likely that every access of a file in git-annex on Windows now goes through UNC-style normalization, allowing long filenames to be used. Assuming that everything in ghc base does it, which I think it does.
So good chance this is fixed now..
I'm informed that this this fix,
git-annex testnow succeeds when run in a deep and long directory path on Windows. It used to fail in that situation. A good indication that long filenames are fully supported now.