This is an appendix to the external special remote protocol.

Some special remotes interface to a key/value datastore using keys that are eg hashes, and it won't make sense for them to implement any of this.

When a special remote interfaces with something that looks like a directory of files, it can implement either or both of the interfaces described here.

The export interface can be implemented to allow the git-annex-export command to be used to store file trees on the special remote.

The import interface can be implemented to allow the git-annex-import command to be used to import file trees from the special remote that are written by some other tool or process.

Note that, when a special remote implements both interfaces, the user will need to choose at initremote time whether to use importtree=yes or exporttree=yes. They won't be allowed to use both, since this interface is not designed to safely allow import and export with the same special remote.

  1. export interface
  2. import interface

export interface

git-annex will use this when the special remote is initialized with exporttree=yes and indicates that it supports exports.

These are requests git-annex sends to the special remote program. Once the special remote has finished performing a request, it should send one of the listed replies. Or, if it does not support a request, it can reply with UNSUPPORTED-REQUEST.

  • EXPORTSUPPORTED
    Used to check if a special remote supports exports. Note that this request may be made before or after PREPARE.
    • EXPORTSUPPORTED-SUCCESS
      Indicates that it makes sense to export to this special remote.
    • EXPORTSUPPORTED-FAILURE
      Indicates that it does not make sense to export to this special remote.
  • EXPORT Name
    Comes immediately before each of the following requests (except for REMOVEEXPORTDIRECTORY), specifying the name of the exported file. It will be in the form of a relative path, and may contain path separators, whitespace, and other special characters.
    No response is made to this message.
    Note that old versions of git-annex had a bug that sometimes prevented sending EXPORT. To avoid being used with such a buggy version of git-annex, send VERSION 2.
  • TRANSFEREXPORT STORE|RETRIEVE Key File
    Requests the transfer of a File on local disk to or from the previously provided EXPORT Name on the special remote.
    Note that it's important that, while a file is being stored, CHECKPRESENTEXPORT not indicate it's present until all the data has been transferred.
    While the transfer is running, the remote can send any number of PROGRESS messages. Once the transfer is complete, it finishes by sending one of these replies:
    • TRANSFER-SUCCESS STORE|RETRIEVE Key
      Indicates the transfer completed successfully.
    • TRANSFER-FAILURE STORE|RETRIEVE Key ErrorMsg
      Indicates the transfer failed.
    • TRANSFER-RETRIEVE-URL Key Url
      Rather than the special remote downloading content from an url itself, this lets it offload that work to git-annex. This response is a protocol extension; it's only safe to send it to git-annex after it sent an EXTENSIONS that included TRANSFER-RETRIEVE-URL.
    • DELEGATE type=value [params]
      Delegate this request to a different type of special remote. See delegate appendix.
  • CHECKPRESENTEXPORT Key
    Requests the remote to check if the previously provided EXPORT Name is present in it.
    • CHECKPRESENT-SUCCESS Key
      Indicates that a content has been positively verified to be present in the remote.
    • CHECKPRESENT-FAILURE Key
      Indicates that a contents has been positively verified to not be present in the remote.
    • CHECKPRESENT-UNKNOWN Key ErrorMsg
      Indicates that it is not currently possible to verify if content is present in the remote. (Perhaps the remote cannot be contacted.)
    • CHECKPRESENT-URL Key Url
      Rather than the special remote checking an url itself, this lets it offload that work to git-annex. This response is a protocol extension; it's only safe to send it to git-annex after it sent an EXTENSIONS that included CHECKPRESENT-URL.
    • DELEGATE type=value [params]
      Delegate this request to a different type of special remote. See delegate appendix.
  • REMOVEEXPORT Key
    Requests the remote to remove content stored by TRANSFEREXPORT with the previously provided EXPORT Name.
    • REMOVE-SUCCESS Key
      Indicates the content has been removed from the remote. May be returned when the content was already not present.
    • REMOVE-FAILURE Key ErrorMsg
      Indicates that the content was unable to be removed from the remote.
    • DELEGATE type=value [params]
      Delegate this request to a different type of special remote. See delegate appendix.
  • REMOVEEXPORTDIRECTORY Directory
    Requests the remote remove an exported directory.
    If the remote does not use directories, or REMOVEEXPORT cleans up directories that are empty, this does not need to be implemented.
    The directory will be in the form of a relative path, and may contain path separators, whitespace, and other special characters.
    Typically the directory will be empty, but it could possibly contain files or other directories, and it's ok to remove those, but better to avoid doing so.
    • REMOVEEXPORTDIRECTORY-SUCCESS
      Indicates that a REMOVEEXPORTDIRECTORY was done successfully.
    • REMOVEEXPORTDIRECTORY-FAILURE
      Indicates that a REMOVEEXPORTDIRECTORY failed for whatever reason. Should not be returned if the directory did not exist.
    • DELEGATE type=value [params]
      Delegate this request to a different type of special remote. See delegate appendix.
  • RENAMEEXPORT Key NewName
    Requests the remote rename a file stored on it from the previously provided EXPORT Name to the NewName. Remotes that support exports but not renaming do not need to implement this.
    • RENAMEEXPORT-SUCCESS Key
      Indicates that a RENAMEEXPORT was done successfully.
    • RENAMEEXPORT-FAILURE Key
      Indicates that a RENAMEEXPORT failed for whatever reason.
    • DELEGATE type=value [params]
      Delegate this request to a different type of special remote. See delegate appendix.

import interface

git-annex will use this interface when the special remote is initialized with importtree=yes and indicates that it supports imports.

git-annex version 10.20260717 was the first to support this.

content identifiers

The special remote needs to have some way to identify a particular version of a file. This is called a ContentIdentifier. A good ContentIdentifier needs to:

  • Be stable, so when a file has not changed, its content identifier remains the same.
  • Change when a file is modified.
  • Be as unique as possible, but not necessarily fully unique. A hash of the content would be ideal. A (size, mtime, inode) tuple is as good a content identifier as git uses in its index. A (size, mtime) tuple will detect most modifications to files, and is usually good enough.
  • Be reasonably short, since it will be stored in the git-annex branch.

It's up to the implementor of a external special remote program what to use for their ContentIdentifier, but not meeting those criteria will lead to unhappy users, and it's better not to implement this interface if you can't do it well.

protocol messages

These are requests that git-annex sends to the special remote program. Once the special remote has finished performing a request, it should send one of the listed replies. Or, if it does not support a request, it can reply with UNSUPPORTED-REQUEST.

  • IMPORTSUPPORTED
    Used to check if a special remote supports imports. Note that this request may be made before or after PREPARE.
    • IMPORTSUPPORTED-SUCCESS
      Indicates that it makes sense to import from this special remote.
    • IMPORTSUPPORTED-FAILURE
      Indicates that it does not make sense to import from this special remote.
    • IMPORTREQUIRED Indicates that this special remote only supports imports. It cannot be used for exports, or used as a key/value store. This allows responding to TRANSFER, CHECKPRESENT, and REMOVE with UNSUPPORTED-REQUEST. When initializing such a special remote, the user will be required to specify the exporttree=yes parameter.
  • LISTIMPORTABLECONTENTS
    Used to get a list of all the files that are stored in the special remote. A block of responses can be made to this, which must always end with LISTIMPORTABLECONTENTS-SUCCESS or `LISTIMPORTABLECONTENTS-FAILURE.
    • IMPORTABLECONTENT Size Name
      A file stored in the special remote. The Size is its size in bytes. The Name is the name of the file on the remote, in the form of a relative path, and may contain path separators, whitespace, and other special characters.
      Always followed by IMPORTABLECONTENTIDENTIFIER.
    • IMPORTABLECONTENTIDENTIFIER ContentIdentifier
      Provide the ContentIdentifier for the previous CONTENT.
    • LISTIMPORTABLECONTENTS-SUCCESS
      Indicates the end of the block of responses and success.
    • LISTIMPORTABLECONTENTS-FAILURE ErrorMsg
      Indicates the end of the block of responses (which may be empty) and failure.
    • DELEGATE type=value [params]
      Delegate this request to a different type of special remote. See delegate appendix.
  • IMPORT Name
    Comes before each of the following requests, specifying the name of the file on the remote. It will be in the form of a relative path, and may contain path separators, whitespace, and other special characters.
    No response is made to this message.
  • RETRIEVEIMPORT File
    Retrieves the content of a file with the previously provided IMPORT Name from the special remote to the File on local disk.
    Note that the special remote can send DOWNLOAD-URL messages without negotiating support for it in the protocol, because versions of git-annex that support this also support DOWNLOAD-URL.
    While the transfer is running, the remote can send any number of PROGRESS messages. Once the transfer is complete, it finishes by sending one of these replies:
    • RETRIEVEIMPORT-SUCCESS Indicates that the retrieve was successful.
    • RETRIEVEIMPORT-FAILURE ErrorMsg
      Indicates the retrieve failed.
    • RETRIEVEIMPORT-URL Url
      Rather than the special remote downloading content from an url itself, this lets it offload that work to git-annex.
    • DELEGATE type=value [params]
      Delegate this request to a different type of special remote. See delegate appendix.
  • CHECKPRESENTIMPORT Key
    Requests the remote to check if the previously provided IMPORT Name is present in it.
    • CHECKPRESENT-SUCCESS Key
      Indicates that a content has been positively verified to be present in the remote.
    • CHECKPRESENT-FAILURE Key
      Indicates that a content has been positively verified to not be present in the remote.
    • CHECKPRESENT-UNKNOWN Key ErrorMsg
      Indicates that it is not currently possible to verify if content is present in the remote. (Perhaps the remote cannot be contacted.)
    • CHECKPRESENT-URL Key Url
      Rather than the special remote checking an url itself, this lets it offload that work to git-annex.
    • DELEGATE type=value [params]
      Delegate this request to a different type of special remote. See delegate appendix.
  • IMPORTKEY Size ContentIdentifier
    Requests the remote generate a git-annex key for content stored on it without downloading that content.
    This request is a protocol extension and will only be sent if the special remote sent an EXTENSIONS that included IMPORTKEY.
    Implementing this allows using git-annex import --no-content from the remote. The remote can generate any type of git-annex key that uses a checksum. For example, if a cloud storage service allows querying for a md5, key, it's ok to generate a MD5 key. To generate the key, see key format.
    The previously provided IMPORT Name specifies which file on the remote to generate the key for.
    The Size is provided because the key should include the size of the file.
    The ContentIdentifier is provided in case it can be used to avoid extra work in determining the key. For example, if a remote uses a md5 as the ContentIdentifier in LISTIMPORTABLECONTENTS, it can reuse the md5 to generate the key. It is not necessary to guarantee that the file on the remote still has the same ContentIdentifier.
    • IMPORTKEY-SUCCESS Key
      Responding with a key.
    • IMPORTKEY-FAILURE ErrorMsg
      Indicates that it was not possible to import a key. (Perhaps because the remote cannot be contacted.)
    • IMPORTKEY-SKIP
      This makes the file not be included in the imported tree.
    • DELEGATE type=value [params]
      Delegate this request to a different type of special remote. See delegate appendix.

import example

The protocol starts off with VERSION etc as usual, and then git-annex asks the special remote if it supports export and import. In this example it supports both:

VERSION 1
EXTENSIONS INFO
EXTENSIONS
PREPARE
EXPORTSUPPORTED
EXPORTSUPPORTED-SUCCESS
IMPORTSUPPORTED
IMPORTSUPPORTED-SUCCESS

git-annex asks for a list of files stored in the special remote:

LISTIMPORTABLECONTENTS

Which responds:

IMPORTABLECONTENT 100 foo
IMPORTABLECONTENTIDENTIFIER 100 48511528411921470
IMPORTABLECONTENT 200 bar
IMPORTABLECONTENTIDENTIFIER 200 48511528411963410
IMPORTABLECONTENTS-SUCCESS

Next git-annex asks for the content of a file to be retrieved.

IMPORT foo
RETRIEVEIMPORT tmpfile

If the file no longer exists, the response would be:

RETRIEVEIMPORT-FAILURE file not found

If the file exists, the response would be:

RETRIEVEIMPORT-SUCCESS

What if the file exists but has been modified in the time since LISTIMPORTABLECONTENTS? The external special remote still retrieves it. git-annex will then notice that the hash doesn't match.