Please describe the problem.
Happens as a part of our test for datalad-archives external special remote we drop/get the file we just added to git-annex from a tarball.
It works fine on Linux, and from todays fix 8.20201007-144-g7245a9ed5 we no longer stall, and proceed to that drop. But drop fails claiming that the content is locked. I have checked processes - there is no some other git annex hanging around and .lck even doesn't exist before we call drop.
(datalad-3) C:\Users\DataLad\AppData\Local\Temp\datalad_temp_tree_ythgiyjt\a\d>dir ..\..\.git\annex\objects\662\c6f\SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.datc\
Volume in drive C has no label.
Volume Serial Number is AC6C-BABE
Directory of C:\Users\DataLad\AppData\Local\Temp\datalad_temp_tree_ythgiyjt\.git\annex\objects\662\c6f\SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.datc
11/02/2020 05:17 PM <DIR> .
11/02/2020 05:17 PM <DIR> ..
11/02/2020 05:17 PM 3 SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.datc 1 File(s) 3 bytes
2 Dir(s) 23,168,851,968 bytes free
but then it is created by drop:
(datalad-3) C:\Users\DataLad\AppData\Local\Temp\datalad_temp_tree_ythgiyjt\a\d>git annex drop --debug -- a.datc
[2020-11-02 17:21:26.8245443] process [5128] read: git ["--git-dir=..\\..\\.git","--work-tree=..\\..","--literal-pathspecs","symbolic-ref","-q","HEAD"]
[2020-11-02 17:21:26.8393936] process [5128] done ExitSuccess
[2020-11-02 17:21:26.8393936] process [1452] read: git ["--git-dir=..\\..\\.git","--work-tree=..\\..","--literal-pathspecs","show-ref","refs/heads/adjusted/dl-test-branch(unlocked)"]
[2020-11-02 17:21:26.8860693] process [1452] done ExitSuccess
[2020-11-02 17:21:26.8860693] process [2004] read: git ["--git-dir=..\\..\\.git","--work-tree=..\\..","--literal-pathspecs","ls-files","--stage","-z","--","a.datc"]
[2020-11-02 17:21:26.9331701] process [3452] chat: git ["--git-dir=..\\..\\.git","--work-tree=..\\..","--literal-pathspecs","cat-file","--batch-check=%(objectname) %(objecttype) %(objectsize)","--buffer"]
[2020-11-02 17:21:26.9501808] process [4632] chat: git ["--git-dir=..\\..\\.git","--work-tree=..\\..","--literal-pathspecs","cat-file","--batch=%(objectname) %(objecttype) %(objectsize)","--buffer"]
[2020-11-02 17:21:26.9501808] process [2920] read: git ["--git-dir=..\\..\\.git","--work-tree=..\\..","--literal-pathspecs","show-ref","git-annex"]
[2020-11-02 17:21:27.0113712] process [2920] done ExitSuccess
[2020-11-02 17:21:27.0113712] process [976] read: git ["--git-dir=..\\..\\.git","--work-tree=..\\..","--literal-pathspecs","show-ref","--hash","refs/heads/git-annex"]
[2020-11-02 17:21:27.0580403] process [976] done ExitSuccess
[2020-11-02 17:21:27.0580403] process [8] read: git ["--git-dir=..\\..\\.git","--work-tree=..\\..","--literal-pathspecs","log","refs/heads/git-annex..6217b1e92de175374dcb552d2bddee0b785073ce","--pretty=%H","-n1"]
[2020-11-02 17:21:27.1363275] process [8] done ExitSuccess
[2020-11-02 17:21:27.1363275] process [6948] chat: git ["--git-dir=..\\..\\.git","--work-tree=..\\..","--literal-pathspecs","hash-object","-w","--stdin-paths","--no-filters"]
[2020-11-02 17:21:27.1675675] process [1728] chat: git ["--git-dir=..\\..\\.git","--work-tree=..\\..","--literal-pathspecs","cat-file","--batch"]
[2020-11-02 17:21:27.1675675] process [4840] chat: git ["--git-dir=..\\..\\.git","--work-tree=..\\..","--literal-pathspecs","cat-file","--batch-check=%(objectname) %(objecttype) %(objectsize)"]
[2020-11-02 17:21:27.1834634] process [7832] feed: git ["--git-dir=..\\..\\.git","--work-tree=..\\..","--literal-pathspecs","update-index","-z","--index-info"]
[2020-11-02 17:21:27.2146236] process [6244] read: git ["--git-dir=..\\..\\.git","--work-tree=..\\..","--literal-pathspecs","diff-index","--raw","-z","-r","--no-renames","-l0","--cached","refs/heads/git-annex","--"]
[2020-11-02 17:21:27.2769931] process [6244] done ExitSuccess
[2020-11-02 17:21:27.3111859] process [7832] done ExitSuccess
[2020-11-02 17:21:27.3238724] process [5528] chat: git ["--git-dir=..\\..\\.git","--work-tree=..\\..","--literal-pathspecs","cat-file","--batch=%(objectname) %(objecttype) %(objectsize)","--buffer"]
[2020-11-02 17:21:27.3864742] process [8556] chat: git ["--git-dir=..\\..\\.git","--work-tree=..\\..","--literal-pathspecs","check-attr","-z","--stdin","annex.backend","annex.numcopies","annex.largefiles","--"]
[2020-11-02 17:21:27.4331328] process [1100] chat: git ["--git-dir=..\\..\\.git","--work-tree=..\\..","--literal-pathspecs","cat-file","--batch"]
[2020-11-02 17:21:27.4331328] process [3516] chat: git ["--git-dir=..\\..\\.git","--work-tree=..\\..","--literal-pathspecs","cat-file","--batch-check=%(objectname) %(objecttype) %(objectsize)"]
drop a.datc
git-annex: content is locked
failed
[2020-11-02 17:21:27.5114868] process [1100] done ExitSuccess
[2020-11-02 17:21:27.5114868] process [3516] done ExitSuccess
[2020-11-02 17:21:27.5114868] process [8556] done ExitSuccess
[2020-11-02 17:21:27.5275735] process [6948] done ExitSuccess
[2020-11-02 17:21:27.5422938] process [5528] done ExitSuccess
[2020-11-02 17:21:27.5422938] process done ExitSuccess
[2020-11-02 17:21:27.5422938] process done ExitSuccess
[2020-11-02 17:21:27.5422938] process [4632] done ExitSuccess
[2020-11-02 17:21:27.5422938] process [3452] done ExitSuccess
[2020-11-02 17:21:27.5422938] process [2004] done ExitSuccess
[2020-11-02 17:21:27.5422938] process done ExitSuccess
git-annex: drop: 1 failed
(datalad-3) C:\Users\DataLad\AppData\Local\Temp\datalad_temp_tree_ythgiyjt\a\d>dir ..\..\.git\annex\objects\662\c6f\SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.datc\
Volume in drive C has no label.
Volume Serial Number is AC6C-BABE
Directory of C:\Users\DataLad\AppData\Local\Temp\datalad_temp_tree_ythgiyjt\.git\annex\objects\662\c6f\SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.datc
11/02/2020 05:21 PM <DIR> .
11/02/2020 05:21 PM <DIR> ..
11/02/2020 05:17 PM 3 SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.datc
11/02/2020 05:21 PM 0 SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.datc.lck
2 File(s) 3 bytes
2 Dir(s) 23,168,679,936 bytes free
so it seems that drop somehow steps on its own feet and gets freaked out.
Here is also some dump from procmon captured during drop, and filtered on this lck file name
5:07:49.2392479 PM git-annex.exe 2144 CreateFile C:\Users\DataLad\AppData\Local\Temp\datalad_temp_tree_j7pvlyz2\.git\annex\objects\662\c6f\SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.datc\SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.datc.lck SUCCESS Desired Access: Generic Read/Write, Disposition: OpenIf, Options: Synchronous IO Non-Alert, Non-Directory File, Attributes: N, ShareMode: Read, Write, Delete, AllocationSize: 0, OpenResult: Opened File System 8228 8664
5:07:49.2393198 PM git-annex.exe 2144 QueryInformationVolume C:\Users\DataLad\AppData\Local\Temp\datalad_temp_tree_j7pvlyz2\.git\annex\objects\662\c6f\SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.datc\SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.datc.lck SUCCESS VolumeCreationTime: 10/22/2020 2:55:22 PM, VolumeSerialNumber: AC6C-BABE, SupportsObjects: True, VolumeLabel: File System 8228 8664
5:07:49.2393295 PM git-annex.exe 2144 QueryAllInformationFile C:\Users\DataLad\AppData\Local\Temp\datalad_temp_tree_j7pvlyz2\.git\annex\objects\662\c6f\SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.datc\SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.datc.lck BUFFER OVERFLOW CreationTime: 11/2/2020 5:02:41 PM, LastAccessTime: 11/2/2020 5:04:56 PM, LastWriteTime: 11/2/2020 5:04:56 PM, ChangeTime: 11/2/2020 5:04:56 PM, FileAttributes: A, AllocationSize: 0, EndOfFile: 0, NumberOfLinks: 1, DeletePending: False, Directory: False, IndexNumber: 0x2e000000015258, EaSize: 0, Access: Generic Read/Write, Position: 0, Mode: Synchronous IO Non-Alert, AlignmentRequirement: Word File System 8228 8664
5:07:49.2393613 PM git-annex.exe 2144 QueryInformationVolume C:\Users\DataLad\AppData\Local\Temp\datalad_temp_tree_j7pvlyz2\.git\annex\objects\662\c6f\SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.datc\SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.datc.lck SUCCESS VolumeCreationTime: 10/22/2020 2:55:22 PM, VolumeSerialNumber: AC6C-BABE, SupportsObjects: True, VolumeLabel: File System 8228 8664
5:07:49.2393740 PM git-annex.exe 2144 QueryAllInformationFile C:\Users\DataLad\AppData\Local\Temp\datalad_temp_tree_j7pvlyz2\.git\annex\objects\662\c6f\SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.datc\SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.datc.lck BUFFER OVERFLOW CreationTime: 11/2/2020 5:02:41 PM, LastAccessTime: 11/2/2020 5:04:56 PM, LastWriteTime: 11/2/2020 5:04:56 PM, ChangeTime: 11/2/2020 5:04:56 PM, FileAttributes: A, AllocationSize: 0, EndOfFile: 0, NumberOfLinks: 1, DeletePending: False, Directory: False, IndexNumber: 0x2e000000015258, EaSize: 0, Access: Generic Read/Write, Position: 0, Mode: Synchronous IO Non-Alert, AlignmentRequirement: Word File System 8228 8664
5:07:49.2394246 PM git-annex.exe 2144 QueryStandardInformationFile C:\Users\DataLad\AppData\Local\Temp\datalad_temp_tree_j7pvlyz2\.git\annex\objects\662\c6f\SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.datc\SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.datc.lck SUCCESS AllocationSize: 0, EndOfFile: 0, NumberOfLinks: 1, DeletePending: False, Directory: False File System 8228 8664
5:07:49.2394500 PM git-annex.exe 2144 SetEndOfFileInformationFile C:\Users\DataLad\AppData\Local\Temp\datalad_temp_tree_j7pvlyz2\.git\annex\objects\662\c6f\SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.datc\SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.datc.lck SUCCESS EndOfFile: 0 File System 8228 8664
5:07:49.2395339 PM git-annex.exe 2144 SetAllocationInformationFile C:\Users\DataLad\AppData\Local\Temp\datalad_temp_tree_j7pvlyz2\.git\annex\objects\662\c6f\SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.datc\SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.datc.lck SUCCESS AllocationSize: 0 File System 8228 8664
5:07:49.2395801 PM git-annex.exe 2144 CloseFile C:\Users\DataLad\AppData\Local\Temp\datalad_temp_tree_j7pvlyz2\.git\annex\objects\662\c6f\SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.datc\SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.datc.lck SUCCESS File System 8228 8664
so it seems to happily create that file.
hehehey -- I think I got the reason at least!
so we exceeded the volume of magical dust windows could swallow without it sticking out somewhere... If I change backend to MD5E we do not observe that issue.
It seems that Utility.LockFile.Windows.openLock must be returning Nothing for this message to be displayed.
Which it does when CreateFile returns
INVALID_HANDLE_VALUE. Which it makes sense it would do for a filename that's too long. Except that's taken to mean the locking failed due to it being locked.(So it seems createfile with the too-long filename is creating the file but then failing that way. Which is weird.)
It might be possible to use windows's
GetLastErrorto find out that it failed due to length, but the API docs don't seem to say what the error value is in that case.Normally ghc modifies filenames on windows to not use the compatability layer that has this filename length limit. But since this is using the low-level CreateFile, that does not happen here.
The code that does that is not exposed (
create_device_namein base's cbits/fs.c) It's basically a matter of prepending\\?\to the path, but it also has to be made absolute and cannot contain '/'.I've implemented something similar in git-annex, which I hope will solve this. I have not tried it on windows yet so leaving the bug open for confirmation.