Fails to drop key on windows (Access denied)

Under very particular conditions a git annex drop --force --all fails in a bare repository on windows with:

git-annex: .\annex\objects\3f7\4a3\XDLRA--refs\XDLRA--refs: DeleteFile "\\\\?\\<BASEDIR>\\<REPODIR>\\annex\\objects\\3f7\\4a3\\XDLRA--refs\\XDLRA--refs": permission denied (Access is denied.)

(<BASEDIR> and <REPODIR> are put by me to improve readability here and below).

The key reported as "access denied" is in fact present and has the "read-only" flag set.

What makes this strange is that I can reliably trigger this error, or avoid it, with small differences in the command sequence -- differences that, to me, should not impact the ability to drop a key.

Below I am posting two complete command sequences (and the diff between them) for triggering and not triggering the error. I edited the logs to make all matching paths uniform, for readability again. The logs are produced by the same unittest under the same runtime conditions. Both logs start by initializing a git repo in a previously non-existing directory. In both cases a type=directory special remote is initialized, but once with exporttree=yes and once without. The only other difference is that in the exporttree=yes case a custom tree is written to the repository (without creating a commit or advancing a branch), and a "fake" export.log record is created that claims that this custom tree was exported from "here" to the initialized directory special remote (this change is committed to the git-annex branch). Afterwards the sequence is identical again.

The failure occurs in the simple case, without the fake-export, and only on windows. Both scenarios work on linux and macos.

I am using git annex version 8.20211117-gc3af94eff and git version 2.34.1.windows.1 on an up-to-date win10 box. But I am also seeing this behavior on a windows appveyor CI runner (windows 10.0.17763) with 2.33.1.windows.1.

Please provide any additional information below.

# drop-failure

[DBG] Initialize empty Git repository at '<BASEDIR>\<REPODIR>' ['--bare']
[DBG] Run ['git', '-c', 'diff.ignoreSubmodules=none', '-C', '<BASEDIR>\\<REPODIR>', 'init', '--bare'] (cwd=<BASEDIR>\<REPODIR>)
[DBG] Run ['git', 'config', '-z', '-l', '--show-origin'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
[DBG] Run ['git', '-c', 'diff.ignoreSubmodules=none', 'config', 'annex.private', 'true'] (cwd=<BASEDIR>\<REPODIR>)
[DBG] Run ['git', '-c', 'diff.ignoreSubmodules=none', 'annex', 'init'] (cwd=<BASEDIR>\<REPODIR>)
[Level 11] stderr|   Detected a filesystem without fifo support.
[Level 11] stderr|   Disabling ssh connection caching.
[Level 11] stderr|   Detected a crippled filesystem.
[DBG] Run ['git', 'config', '-z', '-l', '--show-origin'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
[DBG] Run ['git', 'annex', 'version', '--raw'] (cwd=None) Finished with status 0
[DBG] Run ['git', '-c', 'diff.ignoreSubmodules=none', 'annex', 'initremote', 'origin', 'type=directory', 'directory=C:\\REMOTE', 'encryption=none', '-c', 'annex.dotfiles=true'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
[DBG] Run ['git', 'config', '-z', '-l', '--show-origin'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
[DBG] Run ['git', '-c', 'diff.ignoreSubmodules=none', 'cat-file', 'blob', 'git-annex:remote.log'] (cwd=<BASEDIR>\<REPODIR>)
[DBG] Run ['git', '-c', 'diff.ignoreSubmodules=none', 'annex', 'fsck', '-f', 'origin', '--fast', '--key', 'XDLRA--refs', '-c', 'annex.dotfiles=true'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
[DBG] Run ['git', '-c', 'diff.ignoreSubmodules=none', 'annex', 'drop', '--force', '--key', 'XDLRA--refs', '-c', 'annex.dotfiles=true'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
[DBG] Run ['git', '-c', 'diff.ignoreSubmodules=none', 'annex', 'get', '--key', 'XDLRA--refs', '-c', 'annex.dotfiles=true'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
[DBG] Run ['git', '-c', 'diff.ignoreSubmodules=none', 'annex', 'contentlocation', 'XDLRA--refs', '-c', 'annex.dotfiles=true'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
[DBG] Run ['git', '-c', 'diff.ignoreSubmodules=none', 'annex', 'drop', '--force', '--all', '-c', 'annex.dotfiles=true'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 1
fatal: CommandError(CommandError: '"git" "-c" "diff.ignoreSubmodules=none" "annex" "drop" "--force" "--all" "-c" "annex.dotfiles=true"' failed with exitcode 1 under <BASEDIR>\<REPODIR> [out: 'drop XDLRA--refs
failed
drop XDLRA--refs
failed'] [err: 'git-annex: .\annex\objects\3f7\4a3\XDLRA--refs\XDLRA--refs: DeleteFile "\\\\?\\<BASEDIR>\\<REPODIR>\\annex\\objects\\3f7\\4a3\\XDLRA--refs\\XDLRA--refs": permission denied (Access is denied.)
git-annex: .\annex\objects\3f7\4a3\XDLRA--refs\XDLRA--refs: DeleteFile "\\\\?\\<BASEDIR>\\<REPODIR>\\annex\\objects\\3f7\\4a3\\XDLRA--refs\\XDLRA--refs": permission denied (Access is denied.)
drop: 2 failed'])

# drop-success / exporttree=yes

[DBG] Initialize empty Git repository at '<BASEDIR>\<REPODIR>' ['--bare']
[DBG] Run ['git', '-c', 'diff.ignoreSubmodules=none', '-C', '<BASEDIR>\\<REPODIR>', 'init', '--bare'] (cwd=<BASEDIR>\<REPODIR>)
[DBG] Run ['git', 'config', '-z', '-l', '--show-origin'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
[DBG] Run ['git', '-c', 'diff.ignoreSubmodules=none', 'config', 'annex.private', 'true'] (cwd=<BASEDIR>\<REPODIR>)
[DBG] Run ['git', '-c', 'diff.ignoreSubmodules=none', 'annex', 'init'] (cwd=<BASEDIR>\<REPODIR>)
[Level 11] stderr|   Detected a filesystem without fifo support.
[Level 11] stderr|   Disabling ssh connection caching.
[Level 11] stderr|   Detected a crippled filesystem.
[DBG] Run ['git', 'config', '-z', '-l', '--show-origin'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
[DBG] Run ['git', 'annex', 'version', '--raw'] (cwd=None) Finished with status 0
[DBG] Run ['git', '-c', 'diff.ignoreSubmodules=none', 'annex', 'initremote', 'origin', 'type=directory', 'directory=C:\\REMOTE', 'encryption=none', 'exporttree=yes', '-c', 'annex.dotfiles=true'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
[DBG] Run ['git', 'config', '-z', '-l', '--show-origin'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
[DBG] Run ['git', 'hash-object', '-w', '--stdin'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
[DBG] Run ['git', 'update-index', '--add', '--cacheinfo', '120000', 'a4725c5396b9e1abbb67ad1531b30ba5902ccf02', '.datalad/dotgit/refs'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
[DBG] Run ['git', 'hash-object', '-w', '--stdin'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
[DBG] Run ['git', 'update-index', '--add', '--cacheinfo', '120000', '6e67546efb8b9f1a0ded02f07221586c6a93f292', '.datalad/dotgit/repo.zip'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
[DBG] Run ['git', 'write-tree'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
[DBG] Run ['git', 'hash-object', '-w', '--stdin'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
[DBG] Run ['git', 'read-tree', 'git-annex'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
[DBG] Run ['git', 'update-index', '--add', '--cacheinfo', '100644', '8db4da90bf1b3cd4e9d0b6d079532696e71c1998', 'export.log'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
[DBG] Run ['git', 'write-tree'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
[DBG] Run ['git', 'commit-tree', '-m', 'Fake export', '-p', 'git-annex', '0c8af6e758e0d66d3d0d041c7f3f73f79f0dcab9'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
[DBG] Run ['git', '-c', 'diff.ignoreSubmodules=none', 'update-ref', 'refs/heads/git-annex', '9dbd3377a4cca0cded93986a83bb025a0ac15b66'] (cwd=<BASEDIR>\<REPODIR>)
[DBG] Run ['git', '-c', 'diff.ignoreSubmodules=none', 'cat-file', 'blob', 'git-annex:remote.log'] (cwd=<BASEDIR>\<REPODIR>)
[DBG] Run ['git', '-c', 'diff.ignoreSubmodules=none', 'annex', 'fsck', '-f', 'origin', '--fast', '--key', 'XDLRA--refs', '-c', 'annex.dotfiles=true'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
[DBG] Run ['git', '-c', 'diff.ignoreSubmodules=none', 'annex', 'drop', '--force', '--key', 'XDLRA--refs', '-c', 'annex.dotfiles=true'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
[DBG] Run ['git', '-c', 'diff.ignoreSubmodules=none', 'annex', 'get', '--key', 'XDLRA--refs', '-c', 'annex.dotfiles=true'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
[DBG] Run ['git', '-c', 'diff.ignoreSubmodules=none', 'annex', 'contentlocation', 'XDLRA--refs', '-c', 'annex.dotfiles=true'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
[DBG] Run ['git', '-c', 'diff.ignoreSubmodules=none', 'annex', 'drop', '--force', '--all', '-c', 'annex.dotfiles=true'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
[DATALAD-ANNEX] drop XDLRA--refs ok

And here is the diff between the two sequences (failure(-) vs success(+)

--- annex_remote.txt    2022-01-19 21:25:57.025990250 +0100
+++ export_remote.txt   2022-01-19 21:29:57.262464761 +0100
@@ -8,17 +8,23 @@
 [Level 11] stderr|   Detected a crippled filesystem.
 [DBG] Run ['git', 'config', '-z', '-l', '--show-origin'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
 [DBG] Run ['git', 'annex', 'version', '--raw'] (cwd=None) Finished with status 0
-[DBG] Run ['git', '-c', 'diff.ignoreSubmodules=none', 'annex', 'initremote', 'origin', 'type=directory', 'directory=C:\\REMOTE', 'encryption=none', '-c', 'annex.dotfiles=true'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
+[DBG] Run ['git', '-c', 'diff.ignoreSubmodules=none', 'annex', 'initremote', 'origin', 'type=directory', 'directory=C:\\REMOTE', 'encryption=none', 'exporttree=yes', '-c', 'annex.dotfiles=true'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
 [DBG] Run ['git', 'config', '-z', '-l', '--show-origin'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
+[DBG] Run ['git', 'hash-object', '-w', '--stdin'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
+[DBG] Run ['git', 'update-index', '--add', '--cacheinfo', '120000', 'a4725c5396b9e1abbb67ad1531b30ba5902ccf02', '.datalad/dotgit/refs'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
+[DBG] Run ['git', 'hash-object', '-w', '--stdin'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
+[DBG] Run ['git', 'update-index', '--add', '--cacheinfo', '120000', '6e67546efb8b9f1a0ded02f07221586c6a93f292', '.datalad/dotgit/repo.zip'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
+[DBG] Run ['git', 'write-tree'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
+[DBG] Run ['git', 'hash-object', '-w', '--stdin'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
+[DBG] Run ['git', 'read-tree', 'git-annex'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
+[DBG] Run ['git', 'update-index', '--add', '--cacheinfo', '100644', '8db4da90bf1b3cd4e9d0b6d079532696e71c1998', 'export.log'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
+[DBG] Run ['git', 'write-tree'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
+[DBG] Run ['git', 'commit-tree', '-m', 'Fake export', '-p', 'git-annex', '0c8af6e758e0d66d3d0d041c7f3f73f79f0dcab9'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
+[DBG] Run ['git', '-c', 'diff.ignoreSubmodules=none', 'update-ref', 'refs/heads/git-annex', '9dbd3377a4cca0cded93986a83bb025a0ac15b66'] (cwd=<BASEDIR>\<REPODIR>)
 [DBG] Run ['git', '-c', 'diff.ignoreSubmodules=none', 'cat-file', 'blob', 'git-annex:remote.log'] (cwd=<BASEDIR>\<REPODIR>)
 [DBG] Run ['git', '-c', 'diff.ignoreSubmodules=none', 'annex', 'fsck', '-f', 'origin', '--fast', '--key', 'XDLRA--refs', '-c', 'annex.dotfiles=true'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
 [DBG] Run ['git', '-c', 'diff.ignoreSubmodules=none', 'annex', 'drop', '--force', '--key', 'XDLRA--refs', '-c', 'annex.dotfiles=true'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
 [DBG] Run ['git', '-c', 'diff.ignoreSubmodules=none', 'annex', 'get', '--key', 'XDLRA--refs', '-c', 'annex.dotfiles=true'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
 [DBG] Run ['git', '-c', 'diff.ignoreSubmodules=none', 'annex', 'contentlocation', 'XDLRA--refs', '-c', 'annex.dotfiles=true'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
-[DBG] Run ['git', '-c', 'diff.ignoreSubmodules=none', 'annex', 'drop', '--force', '--all', '-c', 'annex.dotfiles=true'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 1
-fatal: CommandError(CommandError: '"git" "-c" "diff.ignoreSubmodules=none" "annex" "drop" "--force" "--all" "-c" "annex.dotfiles=true"' failed with exitcode 1 under <BASEDIR>\<REPODIR> [out: 'drop XDLRA--refs
-failed
-drop XDLRA--refs
-failed'] [err: 'git-annex: .\annex\objects\3f7\4a3\XDLRA--refs\XDLRA--refs: DeleteFile "\\\\?\\<BASEDIR>\\<REPODIR>\\annex\\objects\\3f7\\4a3\\XDLRA--refs\\XDLRA--refs": permission denied (Access is denied.)
-git-annex: .\annex\objects\3f7\4a3\XDLRA--refs\XDLRA--refs: DeleteFile "\\\\?\\<BASEDIR>\\<REPODIR>\\annex\\objects\\3f7\\4a3\\XDLRA--refs\\XDLRA--refs": permission denied (Access is denied.)
-drop: 2 failed'])
+[DBG] Run ['git', '-c', 'diff.ignoreSubmodules=none', 'annex', 'drop', '--force', '--all', '-c', 'annex.dotfiles=true'] (cwd=<BASEDIR>\<REPODIR>) Finished with status 0
+[DATALAD-ANNEX] drop XDLRA--refs ok

Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)

This observation comes from an effort to implement a git-remote-helper that uses git-annex to enable fetch/push to and from any location that can be reached by any special remote. The fact that git-annex can be used to build things like this is just crazy cool! Thx much!

RSS Atom

comment 1

I tried to construct a test case that I can from from these transcripts.

key=XDLRA--refs
mkdir $base/repo
cd $base/repo
git -c diff.ignoreSubmodules=none -C $base/repo init --bare
git config -z -l --show-origin
git -c diff.ignoreSubmodules=none config annex.private true
git -c diff.ignoreSubmodules=none annex init
git config -z -l --show-origin
git annex version --raw
git -c diff.ignoreSubmodules=none annex initremote origin type=directory directory=$remote encryption=none -c annex.dotfiles=true
git config -z -l --show-origin
git -c diff.ignoreSubmodules=none cat-file blob git-annex:remote.log
git -c diff.ignoreSubmodules=none annex fsck -f origin --fast --key $key -c annex.dotfiles=true
git -c diff.ignoreSubmodules=none annex drop --force --key $key -c annex.dotfiles=true
git -c diff.ignoreSubmodules=none annex get --key $key -c annex.dotfiles=true
git -c diff.ignoreSubmodules=none annex contentlocation $key -c annex.dotfiles=true
git -c diff.ignoreSubmodules=none annex drop --force --all -c annex.dotfiles=true

Now, this would normally fail at the fsck step, since no object file exists for XDLRA--refs. I suppose that you must have pre-populated the special remote with it, somehow, and left that part out. The fsck would then learn that the object does exist there.

It seems to me that how you prepopulated it might be a crucial detail, since that object file gets copied from the special remote and perhaps something to do with its permissions etc are what is later preventing deleting the copy.

You're also using an external backend for whatever reason, and it would simplify the test case if that external backend were not needed and it just used a built-in backend. Using key=WORM--refs seems like it should behave the same, unless your external backend is doing something very strange. If the external backend is necessary to reproduce it, I will need some kind of minimal version of it.

So, I've tried pre-populating the file in the remote like this:

mkdir -p $remote/06b/f75/WORM--refs
echo hi > $remote/06b/f75/WORM--refs/WORM--refs

With that, I've gotten as far as getting the test case to succeed on linux.

But I don't think it's worth the bother of getting a windows environment set up and running these commands on it, without first knowing how you prepopulated the special remote with the object file.

Comment by joey — Wed Jun 22 16:42:25 2022

Remove comment

Possible simplified scenario

Hello, firstly thank you for developing a really useful piece of software. During my initial experimentation I came across what appears to be a variation of this bug, and think I've distilled it to a minimal reproducible scenario.

Initialise in the usual way on an NTFS partition, then add a directory special remote (no encryption, no importtree and no exporttree):

    git init
    git annex init local
    git annex initremote nextdoor type=directory directory=N:\nextdoordir encryption=none

In my case I then added and committed the files locally, then moved them to the directory special remote and back again:

    git annex add .
    git commit --all --message="first commit"
    git annex move . --to nextdoor
    git annex move . --from nextdoor

This completes successfully, however repeating the last two steps a second time triggers the permission denied (Access is denied.) failure at the start of the bug report.

Going through each part step by step:

Since NTFS is designated as a "crippled filesystem", the annexed objects appear to be read-write by default (no ACL modifications, no ReadOnly attribute).
When the files are moved away to the directory special remote (in my test, the same NTFS partition), they pick up a ReadOnly attribute in the new location, so Archive+Compression becomes ReadOnly+Archive+Compression.
When the files are then moved back from the directory special remote, the ReadOnly attribute persists.
Repeating the movement then fails, as the file cannot be dropped locally (the UNC path exists, but DeleteFile fails).

If I remove the ReadOnly attributes and try again, the move away is successful. Similarly if I use a networked ext4 location for the directory special remote (and NTFS locally), the same cycle of success then failure can be observed.

Version information: git git version 2.44.0.windows.1, annex git-annex version: 10.20240130-gad8e32c09d3ec866e0c0654cdcd146bf1aefbc5e (installer from 2024-02-27), Windows 10 22H2

If you require logs or other information, please let me know.

Comment by jasonc — Wed Apr 3 16:48:04 2024