resolvemerge fails when unlocked empty files exist

Please describe the problem.

Causing a file conflict with an unlocked empty file in the repository results in resolvemerge failing.

This can be quirky at times when a resolvemerge unexpectedly fails due to some program inconspicuously creating an empty file, since the repositories I regularly work with have annex.addunlocked set to true.

What steps will reproduce the problem?

With two repositories,
Add an empty file to one repository.
Sync repositories.
Unlock the empty file.
Sync repositories.
Add a file to both repositories with the same name but different contents.
Sync repositories, expecting resolvemerge to automatically fix the conflict, but see it fail instead.

What version of git-annex are you using? On what operating system?

git-annex version: 10.20230321
build flags: Assistant Webapp Pairing Inotify TorrentParser MagicMime Feeds Testsuite S3 WebDAV
dependency versions: aws-0.23 bloomfilter-2.0.1.0 cryptonite-0.30 DAV-1.3.4 feed-1.3.2.1 ghc-9.2.5 http-client-0.7.13.1 persistent-sqlite-2.13.1.0 torrent-10000.1.1 uuid-1.3.15 yesod-1.6.2.1
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL X*
remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs httpalso borg hook external
operating system: linux x86_64
supported repository versions: 8 9 10
upgrade supported from repository versions: 0 1 2 3 4 5 6 7 8 9 10
local repository version: 10

Running git-annex on Fedora 37, with git-annex installed via Guix.

Please provide any additional information below.

The following script reliably produces the issue on my end:

# [1] Set up two connected repositories
create-repo () {
    git init "$1"
    cd "$1"
    git-annex init "$1"
    cd -
}
create-repo repo1
create-repo repo2
cd repo1
git remote add repo2 ../repo2
cd ../repo2
git remote add repo1 ../repo1
cd ..

# [2-5] Syncing unlocked empty file seems fine
cd repo1
touch emptyfile
git-annex add
git-annex unlock
git-annex sync --content
# Syncing other repository succeeds, albeit with an odd error message
# (probably unrelated since it happens without an emptyfile; different bug?)
cd ../repo2
git-annex sync --content
cd ..

# [6-7] Causing a resolvemerge at this point results in the operation failing
cd repo2
echo repo2 line > testfile
git-annex add
git-annex sync --content
cd ../repo1
echo repo1 line > testfile
git-annex add
# Error here
git-annex sync --content

# resolvemerge works again once emptyfile is locked
git-annex lock
git-annex sync --content

The git-annex sync --content command marked with "Error here" in the script produces the following output:

commit 
[master 5c0fd88] git-annex in repo1
 1 file changed, 1 insertion(+)
 create mode 120000 testfile
ok
merge synced/master 
fatal: stash failed
failed
pull repo2 
From ../repo2
   a951533..20ed66d  git-annex     -> repo2/git-annex
 * [new branch]      master        -> repo2/master
   560d494..69204c4  synced/master -> repo2/synced/master

fatal: stash failed

fatal: stash failed
failed
copy testfile (to repo2...) 
(checksum...) ok
pull repo2 

fatal: stash failed

fatal: stash failed
failed
(recording state in git...)
push repo2 
Enumerating objects: 23, done.
Counting objects: 100% (23/23), done.
Delta compression using up to 4 threads
Compressing objects: 100% (15/15), done.
Writing objects: 100% (18/18), 1.65 KiB | 1.65 MiB/s, done.
Total 18 (delta 4), reused 0 (delta 0), pack-reused 0
To ../repo2
   a951533..34ec971  git-annex -> synced/git-annex
 ! [rejected]        master -> synced/master (non-fast-forward)
error: failed to push some refs to '../repo2'
hint: Updates were rejected because a pushed branch tip is behind its remote
hint: counterpart. Check out this branch and integrate the remote changes
hint: (e.g. 'git pull ...') before pushing again.
hint: See the 'Note about fast-forwards' in 'git push --help' for details.
To ../repo2
 ! [rejected]        master -> master (non-fast-forward)
error: failed to push some refs to '../repo2'
hint: Updates were rejected because the tip of your current branch is behind
hint: its remote counterpart. Integrate the remote changes (e.g.
hint: 'git pull ...') before pushing again.
hint: See the 'Note about fast-forwards' in 'git push --help' for details.

  Pushing to repo2 failed.
failed
sync: 4 failed

Curiously, I only managed to stumble on this using magit with Emacs on my repository.

git status and git-annex status don't output any changes, but for some reason magit reports that there are unstaged changes to remove the empty file symlinks.

For example, this is what magit shows before [6-7] in the script:

Head:     master git-annex in repo1

Unstaged changes (1)
modified   emptyfile
@@ -1 +0,0 @@
-/annex/objects/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Recent commits
eac97cf master synced/master repo2/synced/master git-annex in repo1

This other bug report mentions a different problem but also caused by empty unlocked files; might be related?

Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)

I have been incorporating git-annex into my backup system for the past year now, and it has been absolutely wonderful to work with.

Since this is a git bug, and since I don't think there's a reasonable workaround in git-annex, I'm closing this as forwarded to git. done --Joey

RSS Atom

comment 1

I was able to reproduce the problem using the script.

This does not actually involve resolvemerge at all AFAICS. The pull does not result in the repo getting into a conflicted state, because git merge is failing before that point.

joey@darkstar:~/tmp/c/repo1>git --git-dir=.git --work-tree=. -c merge.directoryRenames=false --literal-pathspecs merge --no-edit refs/remotes/repo2/master
fatal: stash failed

Strange error message... That seems like it could be a bug in git. Certianly an unusual code path in it.

If I disable filter.annex.process and filter.annex.clean, the merge "succeeds":

error: Merging is not possible because you have unmerged files.
hint: Fix them up in the work tree, and then use 'git add/rm <file>'
hint: as appropriate to mark resolution and make a commit.
fatal: Exiting because of an unresolved conflict.

I see that it's asking git-annex to clean emptyfile:

joey@darkstar:~/tmp/c/repo1>GIT_TRACE=1 git --git-dir=.git --work-tree=. -c merge.directoryRenames=false --literal-pathspecs merge --no-edit refs/remotes/repo2/master
16:36:51.406445 git.c:439               trace: built-in: git merge --no-edit refs/remotes/repo2/master
16:36:51.409519 run-command.c:655       trace: run_command: 'git-annex smudge --clean -- '\''emptyfile'\'''
16:36:51.427425 git.c:439               trace: built-in: git config --null --list
16:36:51.432485 git.c:439               trace: built-in: git cat-file '--batch-check=%(objectname) %(objecttype) %(objectsize)'
16:36:51.435775 git.c:439               trace: built-in: git cat-file --batch
16:36:51.444646 git.c:439               trace: built-in: git show-ref --hash refs/annex/last-index
16:36:51.461672 run-command.c:655       trace: run_command: git stash create
16:36:51.464416 git.c:439               trace: built-in: git stash create
16:36:51.465753 run-command.c:655       trace: run_command: 'git-annex smudge --clean -- '\''emptyfile'\'''
16:36:51.475280 git.c:439               trace: built-in: git config --null --list
16:36:51.480984 git.c:439               trace: built-in: git cat-file '--batch-check=%(objectname) %(objecttype) %(objectsize)'
16:36:51.484934 git.c:439               trace: built-in: git cat-file --batch
16:36:51.493965 git.c:439               trace: built-in: git show-ref --hash refs/annex/last-index
16:36:51.498182 git.c:439               trace: built-in: git cat-file '--batch-check=%(objectname) %(objecttype) %(objectsize)' --buffer
16:36:51.499385 git.c:439               trace: built-in: git cat-file '--batch=%(objectname) %(objecttype) %(objectsize)' --buffer
16:36:51.500025 git.c:439               trace: built-in: git diff 06f9e23961782cde731bc7ed3e9fb16a930398e1 --staged --raw -z --no-abbrev -G/annex/objects/ --no-renames --ignore-submodules=all --no-textconv --no-ext-diff
fatal: stash failed

Since it asks git-annex to clean emptyfile twice, I'm guessing that it's cleaning two different versions of the file. Perhaps for the locked and the unlocked versions? (But the locked version is a symlink, so why would it clean it?)

I instrumented git-annex smudge --clean to see what git is feeding it on stdin. In both calls shown above, git-annex is fed "". That seems odd, I'd have expected at least one of them to be an annex pointer file.

At this point, my suspicion is that git is cleaning at least one emptyfile version that is not the one in the working tree. But git-annex smudge --clean assumes it's always being used to clean the one in the working tree. And it has various behaviors based on that assumption (which is necessary for good performance). So it may be outputting something that git is not prepared for somehow, which is causing git merge to fail in this unusual way. But, it does not really seem right to me that any output from the clean filter should cause git merge to fail like this. Still thinking it might be a git bug, or a combination of bugs.

Also, this does only happen when emptyfile is in fact empty. Which is interesting... that should not affect the behavior of git-annex smudge --clean at all.

Comment by joey — Mon May 1 19:50:46 2023

Remove comment

comment 2

I think what needs to be done here is write a simple smudge filter script that mocks git-annex smuge --clean, and see if the problem can be reproduced with that. That should pin down what about the behavior of that might be triggering what problem in git.

Comment by joey — Mon May 1 21:08:48 2023

Remove comment

comment 3

Here's a simplified test case that does not use git-annex sync but still uses git-annex.

#!/bin/sh
git init repo1
cd repo1
git-annex init
git commit --allow-empty -m foo
git remote add repo2 ../repo2
cd ..
git clone repo1 repo2
cd repo2
git remote add repo1 ../repo1
cd ..
cd repo1
touch emptyfile
git-annex add
git-annex unlock
git commit -m add
cd ../repo2
git pull
echo repo2 line > testfile
git-annex add
git commit -m add
cd ../repo1
echo repo1 line > testfile
git-annex add
git commit -m foo
git pull repo2 master

Comment by joey — Tue May 2 16:44:03 2023

Remove comment

comment 4

This is a git bug. The test case below does not use git-annex, but causes the same merge problem.

I've reported the bug to the git developers https://lore.kernel.org/git/ZFFGi9i7JGL9xNx5@kitenet.net/T/#u

#!/bin/bash
setup_smudge_clean() {
    cat <<'EOF' > .git/test-clean
#!/bin/sh
mkdir -p ../content
cat > ../content/new.content
v=$(sha1sum ../content/new.content | cut -d ' ' -f 1)
mv ../content/new.content ../content/$v.content
echo "$v"
EOF
    cat <<'EOF' > .git/test-smudge
#!/bin/sh
v=$(cat)
cat ../content/$v.content
EOF
    chmod +x .git/test-clean
    chmod +x .git/test-smudge
    git config filter.foo.smudge .git/test-smudge
    git config filter.foo.clean .git/test-clean
    echo "* filter=foo" > .git/info/attributes
}

git init repo1
cd repo1
setup_smudge_clean
git commit --allow-empty -m foo
git remote add repo2 ../repo2
cd ..
git clone repo1 repo2
cd repo2
git remote add repo1 ../repo1
setup_smudge_clean
cd ..

cd repo1
touch emptyfile
git add emptyfile
git commit -m add
cd ../repo2
git pull

echo foo > testfile
git add testfile
git commit -m foo
cd ../repo1
echo bar > testfile
git add testfile
git commit -m bar
git pull repo2 master

Comment by joey — Tue May 2 16:56:12 2023

Remove comment

comment 5

I think I have been hit by this bug as well with getting "fatal: stash failed" after importing some files and trying to sync.

Indeed, temporarily deactivating filter.annex.process and filter.annex.clean allowed to sync.

And the problem didn't reappear after so far. Is there a reliable workaround (the bug report seem to have no responses?).

git version 2.43.0 git-annex version: 10.20230926-g4ac2758ba589562e427a66437b9fdcd5172357e1