get from sameas remote with importtree=yes fails

Please describe the problem in your own words.

I have a directory special remote with importtree=yes for which I have ran a git annex import to make the files stored there known to git-annex. The same directory is also available via ssh, so I have added another rsync special remote with importtree=yes and --sameas= the directory remote. The content tracking properly shows that they both have the same keys available. The issue is that get'ing a key from the rsync remote fails with the error message "no content identifier is recorded, unable to retrieve".

What steps will reproduce the problem?

mkdir importdir
echo test1 > importdir/test1.txt
echo test2 > importdir/test2.txt
mkdir test-sameas-importtree
cd test-sameas-importtree/
git init
git annex init
git annex initremote importdir type=directory directory=../importdir importtree=yes encryption=none
git annex import importdir --from importdir
git annex drop --all --force
git annex get --key SHA256E-s6--634b027b1b69e1242d40d53e312b3b4ac7710f55be81f289b549446ef6778bee.txt
git annex initremote --sameas=importdir sameas-importdir type=directory directory=../importdir importtree=yes encryption=none
git annex findkeys --in sameas-importdir
git annex get --key SHA256E-s6--7d6fd7774f0d87624da6dcf16d0d3d104c3191e771fbe2f39c86aed4b2bf1a0f.txt --from sameas-importdir
git annex get --key SHA256E-s6--7d6fd7774f0d87624da6dcf16d0d3d104c3191e771fbe2f39c86aed4b2bf1a0f.txt --from importdir

(full transcript of this sequence with outputs is below)

What version of git-annex are you using? On what operating system?

$ git annex version
git-annex version: 10.20260717-g0c917920c80ab1e8cc3d8f5886537708949e1659
build flags: Assistant Webapp Inotify DBus DesktopNotify TorrentParser MagicMime Benchmark Feeds Testsuite S3 WebDAV Servant OsPath Botan Blake3 XXH3
dependency versions: aws-0.24.4 bloomfilter-2.0.1.3 crypton-1.0.4 DAV-1.3.4 feed-1.3.2.1 ghc-9.10.3 http-client-0.7.19 torrent-10000.1.3 uuid-1.3.16 yesod-1.6.2.1
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 BLAKE3_256E BLAKE3_256 XXH3E XXH3 SHA1E SHA1 MD5E MD5 WORM URL GITBUNDLE GITMANIFEST VURL X*
remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs httpalso borg rclone hook external compute mask
operating system: linux x86_64
supported repository versions: 8 9 10
upgrade supported from repository versions: 0 1 2 3 4 5 6 7 8 9 10
local repository version: 10

Please provide any additional information below.

# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log

$ mkdir importdir
$ echo test1 > importdir/test1.txt
$ echo test2 > importdir/test2.txt
$ mkdir test-sameas-importtree
$ cd test-sameas-importtree/
$ git init
Leeres Git-Repository in /home/icg149/Playground/test-sameas-importtree/.git/ initialisiert
$ git annex init
init  ok
(recording state in git...)
$ git annex initremote importdir type=directory directory=../importdir importtree=yes encryption=none
initremote importdir ok
(recording state in git...)
$ git annex import importdir --from importdir
list importdir ok
import importdir test2.txt 
ok                                
import importdir test1.txt 
ok                                
update refs/remotes/importdir/importdir ok
(recording state in git...)
$ git annex drop --all --force
drop SHA256E-s6--634b027b1b69e1242d40d53e312b3b4ac7710f55be81f289b549446ef6778bee.txt ok
drop SHA256E-s6--7d6fd7774f0d87624da6dcf16d0d3d104c3191e771fbe2f39c86aed4b2bf1a0f.txt ok
(recording state in git...)
$ git annex get --key SHA256E-s6--634b027b1b69e1242d40d53e312b3b4ac7710f55be81f289b549446ef6778bee.txt
get SHA256E-s6--634b027b1b69e1242d40d53e312b3b4ac7710f55be81f289b549446ef6778bee.txt (from importdir...) ok
(recording state in git...)
$ git annex initremote --sameas=importdir sameas-importdir type=directory directory=../importdir importtree=yes encryption=none
initremote sameas-importdir ok
(recording state in git...)
$ git annex findkeys --in sameas-importdir
SHA256E-s6--634b027b1b69e1242d40d53e312b3b4ac7710f55be81f289b549446ef6778bee.txt
SHA256E-s6--7d6fd7774f0d87624da6dcf16d0d3d104c3191e771fbe2f39c86aed4b2bf1a0f.txt
$ git annex get --key SHA256E-s6--7d6fd7774f0d87624da6dcf16d0d3d104c3191e771fbe2f39c86aed4b2bf1a0f.txt --from sameas-importdir
get SHA256E-s6--7d6fd7774f0d87624da6dcf16d0d3d104c3191e771fbe2f39c86aed4b2bf1a0f.txt (from sameas-importdir...) 
  no content identifier is recorded, unable to retrieve
failed
get: 1 failed
[ble: exit 1]
$ git annex get --key SHA256E-s6--7d6fd7774f0d87624da6dcf16d0d3d104c3191e771fbe2f39c86aed4b2bf1a0f.txt --from importdir
get SHA256E-s6--7d6fd7774f0d87624da6dcf16d0d3d104c3191e771fbe2f39c86aed4b2bf1a0f.txt (from importdir...) ok
(recording state in git...)

# End of transcript or log.

Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)

fixed --Joey

RSS Atom

comment 1

This happens because the meaning of a content identifier is dependent on the implementation of the special remote. Using a content identifier generated for a directory special remote with a rsync special remote will not work.

So, for the --sameas remote, the content identifier is accessed using its annex-config-uuid rather than the annex-uuid. And that's why it says no content identifier is recorded, since none has been stored using the annex-config-uuid.

For this to work there would need to be another step taken to record the content identifiers for the --sameas remote. Which could be something like git-annex import from it -- except when I tried doing that, it doesn't record them, because it skips over already imported files.

I doubt that needing a second import step would meet your needs though?

It may be that the best fix for this is to prohibit initializing a --sameas remote with importtree=yes.

Comment by joey — Thu Sep 24 13:45:26 2026

comment 2

A use case for this is having a directory special remote that is --sameas a rsync special remote. git-annex import --fast from the directory special remote imports the tree and that avoids needing to import from the rsync special remote, which would be much slower for a large data set.

A second import step would be acceptable in this situation. The user would need to make sure that the files on the remotes are unchanged from the directory import when running the rsync import. This seems like it could be handled as a special flag to git-annex import.

Comment by joey — Thu Sep 24 14:23:45 2026

comment 3

Following is a small patch that makes import from directory, followed by get from the sameas rsync work.

Note that, import from rsync followed by get from sameas directory won't work with this patch. The reason is that directory special remotes support importree+exporttree. And even if they are only configured with exporttree, they currently verify ContentIdentifiers when retrieiving. And when no ContentIdentifier is known, it will fail.

So, applying this patch seems like it would need remotes like directory to have a separate implementation of importActions, rather than the current way that exportImportActions is used to implement importActions. There are not too many remotes that support importtree+exporttree, so maybe that is worth doing.

The other limitation of the patch is that import from rsync followed by get from sameas directory that is configured with importree+exporttree still won't work. And this approach cannot deal with that case.

diff --git a/Remote/Helper/ExportImport.hs b/Remote/Helper/ExportImport.hs
index 23bd86c1ec..7db9ddebfb 100644
--- a/Remote/Helper/ExportImport.hs
+++ b/Remote/Helper/ExportImport.hs
@@ -454,18 +454,13 @@ adjustExportImport' isexport isimport isexportimport annexobjects r rs gc = do

    retrieveFromImport getlocs ciddbv k af dest p = do
        cids <- getkeycids ciddbv k
-       if not (null cids)
-           then getlocs $ \loc ->
-               -- retrieveImport does not guarantee that
-               -- the file it retrieves has the content
-               -- identifier, so it must be strongly
-               -- verified.
-               stronglyverify $
-                   snd <$> retrieveImport (importActions r) loc cids dest (Left k) p
-           -- In case a content identifier is somehow missing,
-           -- try this instead.
-           else retrieveWithoutContentIdentifier $
-               retrieveFromExport getlocs k af dest p
+       getlocs $ \loc ->
+           -- retrieveImport does not guarantee that
+           -- the file it retrieves has any content
+           -- identifier, so it must be strongly
+           -- verified.
+           stronglyverify $
+               snd <$> retrieveImport (importActions r) loc cids dest (Left k) p

    retrieveWithoutContentIdentifier a
        | isexport = a

Comment by joey — Thu Sep 24 15:11:26 2026

comment 4

Maybe this patch is ok to apply without worrying about those other cases. --sameas is not guaranteed to work for all combinations of special remotes and that's why multiple remotes accessing the same data store documents combinations that work.

So it could just be documented that this one specific combination also works. And it happens to be a useful combination to support. If we later want some other combination like import from rsync followed by get from sameas directory to also work, the extra code can be written then to support it. (By eg making Remote.Directory not implement importActions using exportImportActions, or making its retrieveExportWithContentIdentifier not fail when the [ContentIdentifier] list is empty)