Currently for a special remote to support being configured with exporttree=no importtree=yes, it needs to implement the ImportActions interface, which uses ContentIdentifiers for safety and includes some methods that are only needed for exporttree=yes.
Few special remotes support that interface, and probably a lot of them
just can't; they don't have something that can be used as a ContentIdentifier,
or lack the necessary atomicity properties to implement it safely.
(For example, storeExportWithContentIdentifier has a list of old
ContentIdentifiers that are allowed to be overwritten, and requires
that other content does not get overwritten.)
The external special remote protocol does not support that interface yet, due to its complexity and also because noone has requested it. (There is a draft protocol extension for export and import, see https://git-annex.branchable.com/design/external_special_remote_protocol/export_and_import_appendix/#index2h2) (See also add import tree to external special remote protocol)
A simpler interface that supoorts only importtree=yes without needing to worry about exporttree=yes, could let a lot more special remotes support tree import. (For example import tree from rsync special remote.)
Such a special remote could be populated in any way by something outside
git-annex, and git annex import --from remote would download the content
and generate a remote tracking branch. Once imported, other clones could
use git annex get to download files from the special remote.
Bearing in mind that since something is writing to the special remote, any file on it could be overwritten at any point, so such a get may download the wrong content. (So the remote should have retrievalSecurityPolicy = RetrievalVerifiableKeysSecure to make downloads be verified well enough.)
I said this would not use a ContentIdentifier, but it seems it needs some
simple form of ContentIdentifier, which could be just an mtime
(but mtime or mtime+size is not able to detect swaps of 2 files that share
both; using inode or something like that is better).
Without any ContentIdentifier, it seems that each time
git annex import --from remote is run, it would need to re-download
all files from the remote, because it would have no way of knowing
if it had seen a version of a file before. This ContentIdentifier would
be used only to avoid re-downloading when importing. It would not be used
by any other methods. It could even be a dummy value if re-downloading
every file on import is acceptable.
What is needed in such an interface?
listImportableContents :: Annex (Maybe (ContentIdentifier, ImportableContents ByteSize))
-- Retrieves content from an import location to a file.
-- The content retrieved could be anything; it needs to be
-- strongly verified if this is used to download a particular Key
-- that was at one point stored on the remote, since the content
-- of the remote could change at any time.
-- (The MeterUpdate does not need to be used if it writes
-- sequentially to the file.)
-- Throws exception on failure.
retrieveImport :: ImportLocation -> FilePath -> MeterUpdate -> Annex ()
-- Checks if anything is present on the remote at the specified
-- ImportLocation. It may check the size or other characteristics
-- of the Key, but does not need to guarantee that the content on
-- the remote is the same as the Key's content.
-- Throws an exception if the remote cannot be accessed.
checkPresentImport :: Key -> ImportLocation -> Annex Bool
listImportableContents is unchanged, and checkPresentImport above is identical to checkPresentExport.
retrieveImport is very similar to retrieveExport, except that the content retrieved is not a particular Key, and that it does not provide any Verification.
An alternative to the above retrieveImport would be to use
retrieveExport for getting known keys from the remote, and use
retrieveExportWithContentIdentifier with its current interface for
importing new ContentIdentifiers from the remote. It would not need
to ensure that the file really has the ContentIdentifier, but this leaves
open the possibility of importtree only remotes that do at least some
verification of the ContentIdentifier.
The similarity with interface that we already have suggests that perhaps this does not need changes to Types.Remote to implement. It could be done as a Remote.Helper.SimpleImport that takes those 3 methods and translates them to the current interface. Or by complicating Remote.Helper.ExportImport further.. --Joey
I came to this while considering using git-annex to importtree from rclone.
The use case could be automation of workflow to work with docs on google drive or alike (docflow) or even more so for the con/serve "concept" since people do keep and collaborate via google drives and dropboxes. I do not have (yet) a clear idea on for which funded project this could apply, but it might become relevant/affordable one way or another if we guestimate the amount of work needed. Also the protocol enhancement for import/export-tree already looks quite thought-through.
I think it would actually be better to modify Types.Remote and add a new interface specifically for this, than to try to jam it into an existing interface it was not designed for.
This is implemented at the haskell level. But the external special remote protocol doesn't support it yet.
I have updated export and import appendix to document this new interface. I decided to not try to support a combined exporttree=yes and importtree=yes for external special remotes because noone has ever stepped forward to implement an external special remote that can safely do that. I know it will be much easier and safer to implement importtree only external special remotes.
The external special remote interface for this still needs to be implemented. Work has begun in the
importonly_externalbranch.)The external special remote interface is implemented, so I am closing this.
See new make more special remotes support importtree