This is an appendix to the external special remote protocol.

  1. introduction
  2. protocol overview
  3. notes

introduction

Normally, an external special remote can only be used to do one thing at a time, and when git-annex has concurrency enabled, it will start up multiple processes for the same external special remote.

The ASYNC extension lets a single external special remote process handle multiple concurrent jobs, which can be useful if multiple processes would use too many resources, or if it can be better coordinated using a single process.

protocol overview

As usual, the protocol starts by the external special remote sending the version of the protocol it's using.

VERSION 1

This extension is negotiated by git-annex sending an EXTENSIONS message that includes ASYNC, and the external special remote responding in kind.

EXTENSIONS INFO ASYNC
EXTENSIONS ASYNC

(Older versions of git-annex will not include ASYNC in their extensions list. To support them, it's a good idea for the external special remote to fall back to using the regular protocol.)

Once the extension is negotiated, messages in the protocol are tagged with a job number, by prefixing them with "J n".

As usual, the first message git-annex sends is generally PREPARE, which gets tagged with a job number:

J 1 PREPARE

Rather than just responding PREPARE-SUCCESS, the job number has to be included in the reply:

J 1 PREPARE-SUCCESS

Suppose git-annex wants to make some transfers. It can request several at the same time, using different job numbers:

J 1 TRANSFER RETRIEVE Key1 file1
J 2 TRANSFER RETRIEVE Key2 file2

The special remote can now perform both transfers at the same time. If it sends PROGRESS messages for these transfers, they have to be tagged with the job numbers:

J 1 PROGRESS 10
J 2 PROGRESS 500
J 1 PROGRESS 20

The special remote can also send messages that query git-annex for some information. These messages and the reply will also be tagged with a job number.

J 1 GETCONFIG url
J 3 RETRIEVE Key3 file3
J 1 VALUE http://example.com/

One transfers are done, the special remote sends TRANSFER-SUCCESS tagged with the job number.

J 2 TRANSFER-SUCCESS RETRIEVE Key2
J 1 PROGRESS 100
J 1 TRANSFER-SUCCESS RETRIEVE Key1

Lots of different jobs can be requested at the same time.

J 4 CHECKPRESENT Key3
J 5 CHECKPRESENT Key4
J 6 REMOVE Key5
J 4 CHECKPRESENT-SUCCESS Key3
J 6 REMOVE-SUCCESS Key5
J 5 CHECKPRESENT-FAILURE Key4

notes

There will be one job number for each thread that git-annex runs concurrently, so around the same number as the -J value, although in some cases git-annex does more concurrent operations than the -J value.

PREPARE is sent only once per run of a special remote program, and despite being tagged with a job number, it should prepare the special remote to run that and any other jobs.

ERROR should not be tagged with a job number if either git-annex or the special remote needs to send it.

VERSION, EXTENSIONS and ERROR are the only protocol messages that do not get tagged with a job number.