export tree bug when two files with the same content both should be removed

Please describe the problem.

When exporting commits to S3 with versioning=yes and exporttree=yes, deleted annexed objects are marked as deleted but deleted git objects remain. This prevents a user from retrieving a consistent version of the exporttree commit without using git-annex.

What steps will reproduce the problem?

Create a repository and add annexed and git objects.
Add S3 special remote with exporttree=yes and versioning=yes
Export to the remote.
Remove a git object and an annexed object.
Export again and the annexed object will be removed while the git object remains.

Example repository and publicly readable remote having this issue here:

https://github.com/openneurodatasets/ds001705

Git files in the 1.0.0 tag are still present in the S3 1.0.1 export. sub-000101/ses-baseline/pet/sub-000101_ses-baseline_rec-MLEM_pet.json is an example file not present in 1.0.1 that is still present on S3.

What version of git-annex are you using? On what operating system?

Debian bullseye

git-annex version: 10.20220927-ga92546587
build flags: Assistant Webapp Pairing Inotify DBus DesktopNotify TorrentParser MagicMime Benchmark Feeds Testsuite S3 WebDAV
dependency versions: aws-0.22 bloomfilter-2.0.1.0 cryptonite-0.26 DAV-1.3.4 feed-1.3.0.1 ghc-8.8.4 http-client-0.6.4.1 persistent-sqlite-2.10.6.2 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.1.0
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL X*
remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs httpalso borg hook external
operating system: linux x86_64
supported repository versions: 8 9 10
upgrade supported from repository versions: 0 1 2 3 4 5 6 7 8 9 10
local repository version: 10

Please provide any additional information below.

# git annex export 1.0.0 --to=s3-PUBLIC
export s3-PUBLIC .bidsignore 
ok                                
export s3-PUBLIC .datalad/.gitattributes 
ok                                
export s3-PUBLIC .datalad/config 
ok                                
export s3-PUBLIC .gitattributes 
ok                                
export s3-PUBLIC CHANGES 
ok                                
export s3-PUBLIC README 
ok                                
export s3-PUBLIC dataset_description.json 
ok                                
export s3-PUBLIC sub-000101/ses-baseline/anat/sub-000101_ses-baseline_T1w.nii.gz 
ok                                
export s3-PUBLIC sub-000101/ses-baseline/anat/sub-000101_ses-baseline_label-displacementROI_dseg.nii.gz 
ok                                
export s3-PUBLIC sub-000101/ses-baseline/pet/.bidsignore 
ok                                
export s3-PUBLIC sub-000101/ses-baseline/pet/sub-000101_ses-baseline_K1.nii.gz 
ok                                
export s3-PUBLIC sub-000101/ses-baseline/pet/sub-000101_ses-baseline_Vb.nii.gz 
ok                                
export s3-PUBLIC sub-000101/ses-baseline/pet/sub-000101_ses-baseline_k2.nii.gz 
ok                                
export s3-PUBLIC sub-000101/ses-baseline/pet/sub-000101_ses-baseline_k3.nii.gz 
ok                                
export s3-PUBLIC sub-000101/ses-baseline/pet/sub-000101_ses-baseline_k4.nii.gz 
ok                                
export s3-PUBLIC sub-000101/ses-baseline/pet/sub-000101_ses-baseline_rec-MLEM_pet.json 
ok                                
export s3-PUBLIC sub-000101/ses-baseline/pet/sub-000101_ses-baseline_rec-MLEM_pet.nii.gz 
ok                                
export s3-PUBLIC sub-000101/ses-displaced/anat/sub-000101_ses-displaced_T1w.nii.gz 
ok                                
export s3-PUBLIC sub-000101/ses-displaced/anat/sub-000101_ses-displaced_label-displacementROI_dseg.nii.gz 
ok                                
export s3-PUBLIC sub-000101/ses-displaced/pet/.bidsignore 
ok                                
export s3-PUBLIC sub-000101/ses-displaced/pet/sub-000101_ses-displaced_k3.nii.gz 
ok                                
export s3-PUBLIC sub-000101/ses-displaced/pet/sub-000101_ses-displaced_rec-MLEM_pet.json 
ok                                
export s3-PUBLIC sub-000101/ses-displaced/pet/sub-000101_ses-displaced_rec-MLEM_pet.nii.gz 
ok                                
export s3-PUBLIC sub-000102/ses-baseline/anat/sub-000102_ses-baseline_T1w.nii.gz 
ok                                
export s3-PUBLIC sub-000102/ses-baseline/anat/sub-000102_ses-baseline_label-displacementROI_dseg.nii.gz 
ok                                
export s3-PUBLIC sub-000102/ses-baseline/pet/.bidsignore 
ok                                
export s3-PUBLIC sub-000102/ses-baseline/pet/sub-000102_ses-baseline_K1.nii.gz 
ok                                
export s3-PUBLIC sub-000102/ses-baseline/pet/sub-000102_ses-baseline_Vb.nii.gz 
ok                                
export s3-PUBLIC sub-000102/ses-baseline/pet/sub-000102_ses-baseline_k2.nii.gz 
ok                                
export s3-PUBLIC sub-000102/ses-baseline/pet/sub-000102_ses-baseline_k3.nii.gz 
ok                                
export s3-PUBLIC sub-000102/ses-baseline/pet/sub-000102_ses-baseline_k4.nii.gz 
ok                                
export s3-PUBLIC sub-000102/ses-baseline/pet/sub-000102_ses-baseline_rec-MLEM_pet.json 
ok                                
export s3-PUBLIC sub-000102/ses-baseline/pet/sub-000102_ses-baseline_rec-MLEM_pet.nii.gz 
ok                                
export s3-PUBLIC sub-000102/ses-displaced/anat/sub-000102_ses-displaced_T1w.nii.gz 
ok                                
export s3-PUBLIC sub-000102/ses-displaced/anat/sub-000102_ses-displaced_label-displacementROI_dseg.nii.gz 
ok                                
export s3-PUBLIC sub-000102/ses-displaced/pet/.bidsignore 
ok                                
export s3-PUBLIC sub-000102/ses-displaced/pet/sub-000102_ses-displaced_k3.nii.gz 
ok                                
export s3-PUBLIC sub-000102/ses-displaced/pet/sub-000102_ses-displaced_rec-MLEM_pet.json 
ok                                
export s3-PUBLIC sub-000102/ses-displaced/pet/sub-000102_ses-displaced_rec-MLEM_pet.nii.gz 
ok                                
export s3-PUBLIC sub-000103/ses-baseline/anat/sub-000103_ses-baseline_T1w.nii.gz 
ok                                
export s3-PUBLIC sub-000103/ses-baseline/anat/sub-000103_ses-baseline_label-displacementROI_dseg.nii.gz 
ok                                
export s3-PUBLIC sub-000103/ses-baseline/pet/.bidsignore 
ok                                
export s3-PUBLIC sub-000103/ses-baseline/pet/sub-000103_ses-baseline_K1.nii.gz 
ok                                
export s3-PUBLIC sub-000103/ses-baseline/pet/sub-000103_ses-baseline_Vb.nii.gz 
ok                                
export s3-PUBLIC sub-000103/ses-baseline/pet/sub-000103_ses-baseline_k2.nii.gz 
ok                                
export s3-PUBLIC sub-000103/ses-baseline/pet/sub-000103_ses-baseline_k3.nii.gz 
ok                                
export s3-PUBLIC sub-000103/ses-baseline/pet/sub-000103_ses-baseline_k4.nii.gz 
ok                                
export s3-PUBLIC sub-000103/ses-baseline/pet/sub-000103_ses-baseline_rec-MLEM_pet.json 
ok                                
export s3-PUBLIC sub-000103/ses-baseline/pet/sub-000103_ses-baseline_rec-MLEM_pet.nii.gz 
ok                                
export s3-PUBLIC sub-000103/ses-displaced/anat/sub-000103_ses-displaced_T1w.nii.gz 
ok                                
export s3-PUBLIC sub-000103/ses-displaced/anat/sub-000103_ses-displaced_label-displacementROI_dseg.nii.gz 
ok                                
export s3-PUBLIC sub-000103/ses-displaced/pet/.bidsignore 
ok                                
export s3-PUBLIC sub-000103/ses-displaced/pet/sub-000103_ses-displaced_k3.nii.gz 
ok                                
export s3-PUBLIC sub-000103/ses-displaced/pet/sub-000103_ses-displaced_rec-MLEM_pet.json 
ok                                
export s3-PUBLIC sub-000103/ses-displaced/pet/sub-000103_ses-displaced_rec-MLEM_pet.nii.gz 
ok                                
export s3-PUBLIC sub-000104/ses-baseline/anat/sub-000104_ses-baseline_T1w.nii.gz 
ok                                
export s3-PUBLIC sub-000104/ses-baseline/anat/sub-000104_ses-baseline_label-displacementROI_dseg.nii.gz 
ok                                
export s3-PUBLIC sub-000104/ses-baseline/pet/.bidsignore 
ok                                
export s3-PUBLIC sub-000104/ses-baseline/pet/sub-000104_ses-baseline_K1.nii.gz 
ok                                
export s3-PUBLIC sub-000104/ses-baseline/pet/sub-000104_ses-baseline_Vb.nii.gz 
ok                                
export s3-PUBLIC sub-000104/ses-baseline/pet/sub-000104_ses-baseline_k2.nii.gz 
ok                                
export s3-PUBLIC sub-000104/ses-baseline/pet/sub-000104_ses-baseline_k3.nii.gz 
ok                                
export s3-PUBLIC sub-000104/ses-baseline/pet/sub-000104_ses-baseline_k4.nii.gz 
ok                                
export s3-PUBLIC sub-000104/ses-baseline/pet/sub-000104_ses-baseline_rec-MLEM_pet.json 
ok                                
export s3-PUBLIC sub-000104/ses-baseline/pet/sub-000104_ses-baseline_rec-MLEM_pet.nii.gz 
ok                                
export s3-PUBLIC sub-000104/ses-displaced/anat/sub-000104_ses-displaced_T1w.nii.gz 
ok                                
export s3-PUBLIC sub-000104/ses-displaced/anat/sub-000104_ses-displaced_label-displacementROI_dseg.nii.gz 
ok                                
export s3-PUBLIC sub-000104/ses-displaced/pet/.bidsignore 
ok                                
export s3-PUBLIC sub-000104/ses-displaced/pet/sub-000104_ses-displaced_k3.nii.gz 
ok                                
export s3-PUBLIC sub-000104/ses-displaced/pet/sub-000104_ses-displaced_rec-MLEM_pet.json 
ok                                
export s3-PUBLIC sub-000104/ses-displaced/pet/sub-000104_ses-displaced_rec-MLEM_pet.nii.gz 
ok                                
export s3-PUBLIC sub-000105/ses-baseline/anat/sub-000105_ses-baseline_T1w.nii.gz 
ok                                
export s3-PUBLIC sub-000105/ses-baseline/anat/sub-000105_ses-baseline_label-displacementROI_dseg.nii.gz 
ok                                
export s3-PUBLIC sub-000105/ses-baseline/pet/.bidsignore 
ok                                
export s3-PUBLIC sub-000105/ses-baseline/pet/sub-000105_ses-baseline_K1.nii.gz 
ok                                
export s3-PUBLIC sub-000105/ses-baseline/pet/sub-000105_ses-baseline_Vb.nii.gz 
ok                                
export s3-PUBLIC sub-000105/ses-baseline/pet/sub-000105_ses-baseline_k2.nii.gz 
ok                                
export s3-PUBLIC sub-000105/ses-baseline/pet/sub-000105_ses-baseline_k3.nii.gz 
ok                                
export s3-PUBLIC sub-000105/ses-baseline/pet/sub-000105_ses-baseline_k4.nii.gz 
ok                                
export s3-PUBLIC sub-000105/ses-baseline/pet/sub-000105_ses-baseline_rec-MLEM_pet.json 
ok                                
export s3-PUBLIC sub-000105/ses-baseline/pet/sub-000105_ses-baseline_rec-MLEM_pet.nii.gz 
ok                                
export s3-PUBLIC sub-000105/ses-displaced/anat/sub-000105_ses-displaced_T1w.nii.gz 
ok                                
export s3-PUBLIC sub-000105/ses-displaced/anat/sub-000105_ses-displaced_label-displacementROI_dseg.nii.gz 
ok                                
export s3-PUBLIC sub-000105/ses-displaced/pet/.bidsignore 
ok                                
export s3-PUBLIC sub-000105/ses-displaced/pet/sub-000105_ses-displaced_k3.nii.gz 
ok                                
export s3-PUBLIC sub-000105/ses-displaced/pet/sub-000105_ses-displaced_rec-MLEM_pet.json 
ok                                
export s3-PUBLIC sub-000105/ses-displaced/pet/sub-000105_ses-displaced_rec-MLEM_pet.nii.gz 
ok                                 
(recording state in git...)

# git annex export 1.0.1 --to=s3-PUBLIC
unexport s3-PUBLIC sub-000105/ses-baseline/pet/sub-000105_ses-baseline_K1.nii.gz ok
unexport s3-PUBLIC sub-000103/ses-baseline/pet/sub-000103_ses-baseline_k2.nii.gz ok
unexport s3-PUBLIC sub-000104/ses-baseline/pet/sub-000104_ses-baseline_k3.nii.gz ok
unexport s3-PUBLIC .bidsignore ok
unexport s3-PUBLIC sub-000101/ses-baseline/anat/sub-000101_ses-baseline_T1w.nii.gz ok
rename s3-PUBLIC sub-000101/ses-baseline/anat/sub-000101_ses-baseline_label-displacementROI_dseg.nii.gz -> .git-annex-tmp-content-MD5E-s9793--13db6c96808070211b8987c327240ff8.nii.gz ok
unexport s3-PUBLIC sub-000101/ses-baseline/pet/sub-000101_ses-baseline_Vb.nii.gz ok
unexport s3-PUBLIC sub-000102/ses-baseline/pet/sub-000102_ses-baseline_K1.nii.gz ok
unexport s3-PUBLIC sub-000102/ses-baseline/pet/sub-000102_ses-baseline_Vb.nii.gz ok
unexport s3-PUBLIC sub-000104/ses-baseline/anat/sub-000104_ses-baseline_T1w.nii.gz ok
unexport s3-PUBLIC sub-000104/ses-displaced/pet/sub-000104_ses-displaced_rec-MLEM_pet.nii.gz ok
unexport s3-PUBLIC sub-000105/ses-baseline/pet/sub-000105_ses-baseline_k4.nii.gz ok
unexport s3-PUBLIC sub-000102/ses-baseline/pet/sub-000102_ses-baseline_k3.nii.gz ok
unexport s3-PUBLIC sub-000103/ses-displaced/anat/sub-000103_ses-displaced_T1w.nii.gz ok
unexport s3-PUBLIC sub-000105/ses-displaced/pet/sub-000105_ses-displaced_k3.nii.gz ok
rename s3-PUBLIC sub-000102/ses-displaced/anat/sub-000102_ses-displaced_label-displacementROI_dseg.nii.gz -> .git-annex-tmp-content-MD5E-s9794--419a63ecba6de7235554c1f5a7fbc7d8.nii.gz ok
rename s3-PUBLIC sub-000102/ses-baseline/anat/sub-000102_ses-baseline_label-displacementROI_dseg.nii.gz -> .git-annex-tmp-content-MD5E-s9793--45358ccfe2d8a7455c3202691ee632eb.nii.gz ok
unexport s3-PUBLIC sub-000104/ses-baseline/pet/sub-000104_ses-baseline_rec-MLEM_pet.nii.gz ok
rename s3-PUBLIC sub-000105/ses-baseline/anat/sub-000105_ses-baseline_label-displacementROI_dseg.nii.gz -> .git-annex-tmp-content-MD5E-s9793--508a92de80b90e3030d2e46bb920f051.nii.gz ok
rename s3-PUBLIC sub-000101/ses-displaced/anat/sub-000101_ses-displaced_label-displacementROI_dseg.nii.gz -> .git-annex-tmp-content-MD5E-s9794--512f12576f0f99de20f12aa715000251.nii.gz ok
unexport s3-PUBLIC dataset_description.json ok
unexport s3-PUBLIC sub-000105/ses-displaced/pet/sub-000105_ses-displaced_rec-MLEM_pet.nii.gz ok
unexport s3-PUBLIC CHANGES ok
unexport s3-PUBLIC sub-000105/ses-baseline/pet/sub-000105_ses-baseline_Vb.nii.gz ok
unexport s3-PUBLIC sub-000101/ses-displaced/anat/sub-000101_ses-displaced_T1w.nii.gz ok
unexport s3-PUBLIC sub-000103/ses-baseline/pet/sub-000103_ses-baseline_K1.nii.gz ok
unexport s3-PUBLIC sub-000104/ses-baseline/pet/sub-000104_ses-baseline_K1.nii.gz ok
unexport s3-PUBLIC sub-000103/ses-baseline/pet/sub-000103_ses-baseline_rec-MLEM_pet.nii.gz ok
rename s3-PUBLIC sub-000105/ses-displaced/anat/sub-000105_ses-displaced_label-displacementROI_dseg.nii.gz -> .git-annex-tmp-content-MD5E-s9794--726e1c95511fb7b8bc94f275e9d6e9e7.nii.gz ok
unexport s3-PUBLIC sub-000105/ses-baseline/anat/sub-000105_ses-baseline_T1w.nii.gz ok
unexport s3-PUBLIC sub-000105/ses-displaced/pet/.bidsignore ok
unexport s3-PUBLIC sub-000105/ses-displaced/anat/sub-000105_ses-displaced_T1w.nii.gz ok
unexport s3-PUBLIC sub-000104/ses-baseline/pet/sub-000104_ses-baseline_Vb.nii.gz ok
unexport s3-PUBLIC sub-000103/ses-displaced/pet/sub-000103_ses-displaced_rec-MLEM_pet.nii.gz ok
unexport s3-PUBLIC sub-000102/ses-baseline/pet/sub-000102_ses-baseline_rec-MLEM_pet.nii.gz ok
rename s3-PUBLIC sub-000103/ses-displaced/anat/sub-000103_ses-displaced_label-displacementROI_dseg.nii.gz -> .git-annex-tmp-content-MD5E-s9794--887598ee2277aeebae901fd7d97a7f4c.nii.gz ok
unexport s3-PUBLIC sub-000101/ses-baseline/pet/sub-000101_ses-baseline_k4.nii.gz ok
unexport s3-PUBLIC sub-000105/ses-displaced/pet/sub-000105_ses-displaced_rec-MLEM_pet.json ok
unexport s3-PUBLIC sub-000103/ses-baseline/pet/sub-000103_ses-baseline_Vb.nii.gz ok
unexport s3-PUBLIC sub-000103/ses-baseline/pet/sub-000103_ses-baseline_k3.nii.gz ok
unexport s3-PUBLIC sub-000103/ses-displaced/pet/sub-000103_ses-displaced_k3.nii.gz ok
unexport s3-PUBLIC sub-000103/ses-baseline/pet/sub-000103_ses-baseline_k4.nii.gz ok
unexport s3-PUBLIC sub-000105/ses-baseline/pet/sub-000105_ses-baseline_k3.nii.gz ok
unexport s3-PUBLIC sub-000105/ses-baseline/pet/sub-000105_ses-baseline_rec-MLEM_pet.nii.gz ok
unexport s3-PUBLIC sub-000103/ses-baseline/anat/sub-000103_ses-baseline_T1w.nii.gz ok
unexport s3-PUBLIC sub-000102/ses-displaced/anat/sub-000102_ses-displaced_T1w.nii.gz ok
unexport s3-PUBLIC sub-000105/ses-baseline/pet/sub-000105_ses-baseline_k2.nii.gz ok
unexport s3-PUBLIC sub-000101/ses-displaced/pet/sub-000101_ses-displaced_k3.nii.gz ok
unexport s3-PUBLIC README ok
rename s3-PUBLIC sub-000104/ses-displaced/anat/sub-000104_ses-displaced_label-displacementROI_dseg.nii.gz -> .git-annex-tmp-content-MD5E-s9794--bd645a943d74dda136f4e0c421d49c85.nii.gz ok
unexport s3-PUBLIC sub-000104/ses-displaced/pet/sub-000104_ses-displaced_k3.nii.gz ok
unexport s3-PUBLIC sub-000101/ses-baseline/pet/sub-000101_ses-baseline_K1.nii.gz ok
unexport s3-PUBLIC sub-000101/ses-displaced/pet/sub-000101_ses-displaced_rec-MLEM_pet.nii.gz ok
unexport s3-PUBLIC sub-000102/ses-displaced/pet/sub-000102_ses-displaced_k3.nii.gz ok
unexport s3-PUBLIC sub-000102/ses-baseline/anat/sub-000102_ses-baseline_T1w.nii.gz ok
unexport s3-PUBLIC sub-000101/ses-baseline/pet/sub-000101_ses-baseline_k2.nii.gz ok
unexport s3-PUBLIC sub-000102/ses-baseline/pet/sub-000102_ses-baseline_k4.nii.gz ok
unexport s3-PUBLIC sub-000102/ses-baseline/pet/sub-000102_ses-baseline_k2.nii.gz ok
rename s3-PUBLIC sub-000104/ses-baseline/anat/sub-000104_ses-baseline_label-displacementROI_dseg.nii.gz -> .git-annex-tmp-content-MD5E-s9793--de819fa10f3f6253e4eeec3b77158726.nii.gz ok
unexport s3-PUBLIC sub-000102/ses-displaced/pet/sub-000102_ses-displaced_rec-MLEM_pet.nii.gz ok
unexport s3-PUBLIC sub-000101/ses-baseline/pet/sub-000101_ses-baseline_k3.nii.gz ok
rename s3-PUBLIC sub-000103/ses-baseline/anat/sub-000103_ses-baseline_label-displacementROI_dseg.nii.gz -> .git-annex-tmp-content-MD5E-s9793--e6f2a37df1a0a47325fc9cb065117e0d.nii.gz ok
unexport s3-PUBLIC sub-000104/ses-baseline/pet/sub-000104_ses-baseline_k2.nii.gz ok
unexport s3-PUBLIC sub-000101/ses-baseline/pet/sub-000101_ses-baseline_rec-MLEM_pet.nii.gz ok
unexport s3-PUBLIC sub-000104/ses-baseline/pet/sub-000104_ses-baseline_k4.nii.gz ok
unexport s3-PUBLIC sub-000104/ses-displaced/anat/sub-000104_ses-displaced_T1w.nii.gz ok
rename s3-PUBLIC .git-annex-tmp-content-MD5E-s9793--13db6c96808070211b8987c327240ff8.nii.gz -> derivatives/masks/sub-000101/ses-baseline/sub-000101_ses-baseline_label-displacementROI_dseg.nii.gz ok
rename s3-PUBLIC .git-annex-tmp-content-MD5E-s9794--419a63ecba6de7235554c1f5a7fbc7d8.nii.gz -> derivatives/masks/sub-000102/ses-displaced/sub-000102_ses-displaced_label-displacementROI_dseg.nii.gz ok
rename s3-PUBLIC .git-annex-tmp-content-MD5E-s9793--45358ccfe2d8a7455c3202691ee632eb.nii.gz -> derivatives/masks/sub-000102/ses-baseline/sub-000102_ses-baseline_label-displacementROI_dseg.nii.gz ok
rename s3-PUBLIC .git-annex-tmp-content-MD5E-s9793--508a92de80b90e3030d2e46bb920f051.nii.gz -> derivatives/masks/sub-000105/ses-baseline/sub-000105_ses-baseline_label-displacementROI_dseg.nii.gz ok
rename s3-PUBLIC .git-annex-tmp-content-MD5E-s9794--512f12576f0f99de20f12aa715000251.nii.gz -> derivatives/masks/sub-000101/ses-displaced/sub-000101_ses-displaced_label-displacementROI_dseg.nii.gz ok
rename s3-PUBLIC .git-annex-tmp-content-MD5E-s9794--726e1c95511fb7b8bc94f275e9d6e9e7.nii.gz -> derivatives/masks/sub-000105/ses-displaced/sub-000105_ses-displaced_label-displacementROI_dseg.nii.gz ok
rename s3-PUBLIC .git-annex-tmp-content-MD5E-s9794--887598ee2277aeebae901fd7d97a7f4c.nii.gz -> derivatives/masks/sub-000103/ses-displaced/sub-000103_ses-displaced_label-displacementROI_dseg.nii.gz ok
rename s3-PUBLIC .git-annex-tmp-content-MD5E-s9794--bd645a943d74dda136f4e0c421d49c85.nii.gz -> derivatives/masks/sub-000104/ses-displaced/sub-000104_ses-displaced_label-displacementROI_dseg.nii.gz ok
rename s3-PUBLIC .git-annex-tmp-content-MD5E-s9793--de819fa10f3f6253e4eeec3b77158726.nii.gz -> derivatives/masks/sub-000104/ses-baseline/sub-000104_ses-baseline_label-displacementROI_dseg.nii.gz ok
rename s3-PUBLIC .git-annex-tmp-content-MD5E-s9793--e6f2a37df1a0a47325fc9cb065117e0d.nii.gz -> derivatives/masks/sub-000103/ses-baseline/sub-000103_ses-baseline_label-displacementROI_dseg.nii.gz ok
export s3-PUBLIC .bidsignore 
ok                                
export s3-PUBLIC CHANGES 
ok                                
export s3-PUBLIC LICENSE 
ok                                
export s3-PUBLIC README 
ok                                
export s3-PUBLIC dataset_description.json 
ok                                
export s3-PUBLIC derivatives/masks/dseg.tsv 
ok                                
export s3-PUBLIC participants.json 
ok                                
export s3-PUBLIC participants.tsv 
ok                                
export s3-PUBLIC sub-000101/ses-baseline/anat/sub-000101_ses-baseline_T1w.json 
ok                                
export s3-PUBLIC sub-000101/ses-baseline/anat/sub-000101_ses-baseline_T1w.nii 
ok                                 
export s3-PUBLIC sub-000101/ses-baseline/pet/sub-000101_ses-baseline_pet.json 
ok                                
export s3-PUBLIC sub-000101/ses-baseline/pet/sub-000101_ses-baseline_pet.nii 
ok                                   
export s3-PUBLIC sub-000101/ses-displaced/anat/sub-000101_ses-displaced_T1w.json 
ok                                
export s3-PUBLIC sub-000101/ses-displaced/anat/sub-000101_ses-displaced_T1w.nii 
ok                                 
export s3-PUBLIC sub-000101/ses-displaced/pet/sub-000101_ses-displaced_pet.json 
ok                                
export s3-PUBLIC sub-000101/ses-displaced/pet/sub-000101_ses-displaced_pet.nii 
ok                                   
export s3-PUBLIC sub-000102/ses-baseline/anat/sub-000102_ses-baseline_T1w.json 
ok                                
export s3-PUBLIC sub-000102/ses-baseline/anat/sub-000102_ses-baseline_T1w.nii 
ok                                 
export s3-PUBLIC sub-000102/ses-baseline/pet/sub-000102_ses-baseline_pet.json 
ok                                
export s3-PUBLIC sub-000102/ses-baseline/pet/sub-000102_ses-baseline_pet.nii 
ok                                   
export s3-PUBLIC sub-000102/ses-displaced/anat/sub-000102_ses-displaced_T1w.json 
ok                                
export s3-PUBLIC sub-000102/ses-displaced/anat/sub-000102_ses-displaced_T1w.nii 
ok                                
export s3-PUBLIC sub-000102/ses-displaced/pet/sub-000102_ses-displaced_pet.json 
ok                                
export s3-PUBLIC sub-000102/ses-displaced/pet/sub-000102_ses-displaced_pet.nii 
ok                                   
export s3-PUBLIC sub-000103/ses-baseline/anat/sub-000103_ses-baseline_T1w.json 
ok                                
export s3-PUBLIC sub-000103/ses-baseline/anat/sub-000103_ses-baseline_T1w.nii 
ok                                
export s3-PUBLIC sub-000103/ses-baseline/pet/sub-000103_ses-baseline_pet.json 
ok                                
export s3-PUBLIC sub-000103/ses-baseline/pet/sub-000103_ses-baseline_pet.nii 
ok                                  
export s3-PUBLIC sub-000103/ses-displaced/anat/sub-000103_ses-displaced_T1w.json 
ok                                
export s3-PUBLIC sub-000103/ses-displaced/anat/sub-000103_ses-displaced_T1w.nii 
ok                                
export s3-PUBLIC sub-000103/ses-displaced/pet/sub-000103_ses-displaced_pet.json 
ok                                
export s3-PUBLIC sub-000103/ses-displaced/pet/sub-000103_ses-displaced_pet.nii 
ok                                   
export s3-PUBLIC sub-000104/ses-baseline/anat/sub-000104_ses-baseline_T1w.json 
ok                                
export s3-PUBLIC sub-000104/ses-baseline/anat/sub-000104_ses-baseline_T1w.nii 
ok                                
export s3-PUBLIC sub-000104/ses-baseline/pet/sub-000104_ses-baseline_pet.json 
ok                                
export s3-PUBLIC sub-000104/ses-baseline/pet/sub-000104_ses-baseline_pet.nii 
ok                                   
export s3-PUBLIC sub-000104/ses-displaced/anat/sub-000104_ses-displaced_T1w.json 
ok                                
export s3-PUBLIC sub-000104/ses-displaced/anat/sub-000104_ses-displaced_T1w.nii 
ok                                
export s3-PUBLIC sub-000104/ses-displaced/pet/sub-000104_ses-displaced_pet.json 
ok                                
export s3-PUBLIC sub-000104/ses-displaced/pet/sub-000104_ses-displaced_pet.nii 
ok                                   
export s3-PUBLIC sub-000105/ses-baseline/anat/sub-000105_ses-baseline_T1w.json 
ok                                
export s3-PUBLIC sub-000105/ses-baseline/anat/sub-000105_ses-baseline_T1w.nii 
ok                                
export s3-PUBLIC sub-000105/ses-baseline/pet/sub-000105_ses-baseline_pet.json 
ok                                
export s3-PUBLIC sub-000105/ses-baseline/pet/sub-000105_ses-baseline_pet.nii 
ok                                   
export s3-PUBLIC sub-000105/ses-displaced/anat/sub-000105_ses-displaced_T1w.json 
ok                                
export s3-PUBLIC sub-000105/ses-displaced/anat/sub-000105_ses-displaced_T1w.nii 
ok                                
export s3-PUBLIC sub-000105/ses-displaced/pet/sub-000105_ses-displaced_pet.json 
ok                                
export s3-PUBLIC sub-000105/ses-displaced/pet/sub-000105_ses-displaced_pet.nii 
ok                                   
(recording state in git...)

# End of transcript or log.

Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)

Thank you for all your work making this wonderful tool!

fixed --Joey

RSS Atom

comment 1

I tried making a repository with just 2 files, one in git and one in git-annex, and am unable to reproduce the bug. Here is what git-annex export master --to remote --debug showed when exporting a tree that deleted file "foo" which was a git object:

[2022-11-09 14:16:09.513485291] (Remote.S3) String to sign: "DELETE\n/foo\n\nhost:t-a9b9d406-30e5-41cc-a74c-c5d83b2953fb.s3.amazonaws.com\nx-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\nx-amz-date:20221109T181609Z\n\nhost;x-amz-content-sha256;x-amz-date\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
[2022-11-09 14:16:09.513608493] (Remote.S3) Host: "t-a9b9d406-30e5-41cc-a74c-c5d83b2953fb.s3.amazonaws.com"
[2022-11-09 14:16:09.513697445] (Remote.S3) Path: "/foo"
[2022-11-09 14:16:09.513748086] (Remote.S3) Query string: ""
[2022-11-09 14:16:09.513829584] (Remote.S3) Header: (redacted -- JEH)
[2022-11-09 14:16:09.687814925] (Remote.S3) Response status: Status {statusCode = 204, statusMessage = "No Content"}

The S3 console showed that the file was deleted from the bucket. And as far as the S3 remote implementation is concerned, there should not be anything different between a git object and a git-annex object. At the level of the S3 remote both have a git-annex key that it deletes in the same way.

In your log, the only thing it does with the file is export it, but it does not later unexport it:

$ grep baseline/pet/sub-000101_ses-baseline_rec-MLEM_pet.json Versioned_S3_tree_does_not_unexport_git_objects.mdwn
Git files in the 1.0.0 tag are still present in the S3 1.0.1 export. sub-000101/ses-baseline/pet/sub-000101_ses-baseline_rec-MLEM_pet.json is an example file not present in 1.0.1 that is still present on S3.
export s3-PUBLIC sub-000101/ses-baseline/pet/sub-000101_ses-baseline_rec-MLEM_pet.json 
$

If the bug is that it's somehow failing to try to unexport the file, that should happen independently of the special remote type, so would also happen with a directory special remote. So I tried that:

$ git clone https://github.com/openneurodatasets/ds001705
$ cd ds001705
$ git-annex get --branch=tags/1.0.0
$ git-annex get --branch=tags/1.0.1
$ mkdir ../d
$ git-annex initremote d type=directory directory=../d encryption=none exporttree=yes
$ git-annex export 1.0.0 --to d
$ ls ../d/sub-000101/ses-baseline/pet/sub-000101_ses-baseline_rec-MLEM_pet.json
../d/sub-000101/ses-baseline/pet/sub-000101_ses-baseline_rec-MLEM_pet.json
$ git-annex export 1.0.1 --to d
$ ls ../d/sub-000101/ses-baseline/pet/sub-000101_ses-baseline_rec-MLEM_pet.json
../d/sub-000101/ses-baseline/pet/sub-000101_ses-baseline_rec-MLEM_pet.json

Ok, so, nothing to do with S3 or versioning at all.

Comment by joey — Wed Nov 9 18:16:29 2022

Remove comment

comment 2

Interestingly, it does not happen in simpler situations:

$ git checkout 1.0.0
$ git-annex export 1.0.0 --to d
$ git rm sub-000101/ses-baseline/pet/sub-000101_ses-baseline_rec-MLEM_pet.json
$ git commit -m removed
$ git-annex export HEAD --to d
unexport d sub-000101/ses-baseline/pet/sub-000101_ses-baseline_rec-MLEM_pet.json ok
$ ls ../d/sub-000101/ses-baseline/pet/sub-000101_ses-baseline_rec-MLEM_pet.json
ls: cannot access '../d/sub-000101/ses-baseline/pet/sub-000101_ses-baseline_rec-MLEM_pet.json': No such file or directory

So something about the diff between 1.0.0 and 1.0.1 is somehow causing the bug..

Comment by joey — Wed Nov 9 19:15:59 2022

comment 3

Ok, the bug is due to 2 files that have the same content.

sub-000101/ses-baseline/pet/sub-000101_ses-baseline_rec-MLEM_pet.json
sub-000101/ses-displaced/pet/sub-000101_ses-displaced_rec-MLEM_pet.json

Both files get deleted. And the bug makes it only pick one of the two files to delete, because it's using a map from key to file and the second file overwrites the first in the map.

So this would also presumably affect annexed files when two have the same content and are being deleted.

Comment by joey — Wed Nov 9 19:41:33 2022

comment 4

Nice, thank you for looking into this, Joey! That makes sense why it would appear to be git files, we tend to have identical duplicate metadata files much more often than duplicate image files within datasets.

Comment by nell — Wed Nov 9 19:51:20 2022