The git-annex P2P protocol is a custom protocol that git-annex uses to communicate between peers.
There's a common line-based serialization of the protocol, but other serializations are also possible. The line-based serialization is spoken by git-annex-shell, and by git-annex over tor. There is also a translation of this protocol to a HTTP API.
One peer is known as the client, and is the peer that initiates the connection and sends commands. The other peer is known as the server, and is the peer that the client connects to. It's possible for two connections to be run at the same time between the same two peers, in different directions.
Errors
Either the client or the server may send an error message at any time.
When the client sends an ERROR, the server will close the connection.
If the server sends an ERROR in response to the client's request, the connection will remain open, and the client can make another request.
ERROR this repository is read-only; write access denied
Authentication
The protocol generally starts with authentication. However, if authentication already occurs on another layer, as is the case with git-annex-shell, authentication will be skipped.
The client starts by sending an authentication command to the server, along with its UUID. The AuthToken is some arbitrary token that has been agreed upon beforehand.
AUTH UUID AuthToken
The server responds with either its own UUID when authentication is successful. Or, it can fail the authentication, and close the connection.
AUTH-SUCCESS UUID
AUTH-FAILURE
Note that authentication does not guarantee that the client is talking to who they expect to be talking to. This, and encryption of the connection, are handled at a lower level.
Protocol version
The default protocol version is 0. The client can choose to negotiate a new version with the server. This must come after any authentication.
The client sends the highest protocol version it supports:
VERSION 4
The server responds with the highest protocol version it supports that is less than or equal to the version the client sent:
VERSION 1
Now both client and server should use version 1.
Cluster cycle prevention
In protocol version 2 and above, immediately after VERSION, the client can send an additional message that is used to prevent cycles when accessing clusters.
BYPASS UUID1 UUID2 ...
The UUIDs are cluster gateways to avoid connecting to when serving a cluster.
The server makes no response to this message.
Binary data
The protocol allows raw binary data to be sent. This is done using a DATA message. In the line-based serialization, this comes on its own line, followed by a newline and the binary data. The Len value tells how many bytes of data to read.
DATA 3
foo
Note that there is no newline after the binary data; the next protocol message will come immediately after it.
If the sender finds itself unable to send as many bytes of data as it promised (perhaps because a file got truncated while it was being sent), its only option is to close the protocol connection.
And if the receiver finds itself unable to receive all the data for some reason (eg, out of disk space), its only option is to close the protocol connection.
Checking if content is present
To check if a key is currently present on the server, the client sends:
CHECKPRESENT Key
The server responds with either SUCCESS or FAILURE.
Locking content
To lock content on the server, preventing it from being removed, the client sends:
LOCKCONTENT Key
The server responds with either SUCCESS or FAILURE. The former indicates the content is locked.
After SUCCESS, the content will remain locked until the client sends its next message, which must be:
UNLOCKCONTENT Key
The server makes no response to that.
If the connection is broken before the client sends UNLOCKCONTENT, the content will remain locked for at least 10 minutes from when the server sent SUCCESS.
Removing content
To remove a key's content from the server, the client sends:
REMOVE Key
The server responds with either SUCCESS or FAILURE.
Note that if the content was not present, SUCCESS will be returned.
In protocol version 2 and above, the server can optionally reply with SUCCESS-PLUS or FAILURE-PLUS. Each has a subsequent list of UUIDs of repositories that the content was removed from.
Removing content before a specified time
This is only available in protocol version 3 and above.
To remove a key's content from the server, but only before a specified time, the client sends:
REMOVE-BEFORE Timestamp Key
The server responds to the message in the same way as to REMOVE.
If the server receives the message at a time after the specified timestamp, the remove must fail. This is used to avoid removing content after a point in time where it is no longer locked in other repostitories.
Getting a timestamp
This is only available in protocol version 3 and above.
To get the current timestamp from the server, the client sends:
GETTIMESTAMP
The server responds with TIMESTAMP followed by its current time, as a number of seconds. Note that this uses a monotonic clock.
Storing content on the server
To store content on the server, the client sends:
PUT AssociatedFile Key
Here AssociatedFile may be the name of a file in the git repository, for information purposes only. Or it can be the empty string. It will always have unix directory separators.
(Note that in the line-based serialization. AssociatedFile may not contain any spaces, since it's not the last token in the line. Use '%' to indicate whitespace.)
The server may respond with ALREADY-HAVE if it already had the content of that key.
In protocol version 2 and above, the server can optionally reply with ALREADY-HAVE-PLUS. The subsequent list of UUIDs are additional UUIDs where the content is stored, in addition to the UUID where the client was going to send it.
When the server wants the client to send the content to it, it replies with:
PUT-FROM Offset
Offset is the number of bytes into the file that the server wants the client to start. This allows resuming transfers.
The client then sends a DATA message with content of the file from the offset to the end of file.
In protocol version 1 and above, after the DATA, the client sends an additional message, to indicate if the content of the file has changed while it was being sent.
INVALID
VALID
If the server successfully receives the data and stores the content, it replies with SUCCESS. Otherwise, FAILURE.
In protocol version 2 and above, the server can optionally reply with SUCCESS-PLUS and a list of UUIDs where the content was stored.
In protocol version 4 and above, rather than sending a DATA message, the client can send the content to the repository in some other way than using the protocol. To indicate that it has sent the content like that, the client sends:
DATA-PRESENT
The server responds to DATA-PRESENT by checking if the content is present in the repository. If so it proceeds the same as if the client had used DATA to send the data and then sent VALID. Eg, it responds with SUCCESS (or FAILURE if it cannot verify that the data is present).
Getting content from the server
To get content from the server, the client sends:
GET Offset AssociatedFile Key
The Offset is the number of bytes into the file that the client wants the server to skip, which allows resuming transfers. See description of AssociatedFile above.
The server then sends a DATA message with the content of the file from the offset to end of file.
In protocol version 1 and above, after the data, the server sends an additional message, to indicate if the content of the file has changed while it was being sent.
INVALID
VALID
The client replies with SUCCESS or FAILURE.
Note that the client responding with SUCCESS does not indicate to the server that it has stored the content. It may receive it and throw it away.
Connection to services
This is used to connect to services like git-upload-pack and git-receive-pack that speak their own protocol.
The client sends a message to request the connection. Service is the name of the service, eg "git-upload-pack".
CONNECT Service
Both client and server may now exchange DATA messages in any order, encapsulating the service's protocol.
When the service exits, the server indicates this by telling the client its exit code.
CONNECTDONE ExitCode
After that, the server closes the connection.
Change notification
The client can request to be notified when a ref in the git repository on the server changes.
NOTIFYCHANGE
The server will block until at least one of the refs changes, and send a list of changed refs.
CHANGED ChangedRefs
For example:
CHANGED refs/heads/master refs/heads/git-annex
Some servers may not support this command.
eh... i look elsewhere for a week and you design another line protocol!
so I guess it's too late to do anything to change this,
but I wanted to share that similar efforts are being done over the
backup software world, in particular in restic,
which is working with the rclone project to implement an abstract
get/pull mechanism to store blobs, a lot like what git-annex needs to
be doing.
they wrote this using a binary protocol for speed (it's basically RPC at this point) and I encouraged them to at least use a standard one (they use protobufs + HTTP2 AKA gRPC, iirc, and it works over stdin/out). you might find the full thread interesting... it would be great if git-annex would support this natively instead of rolling its own protocol, because it would mean it could talk with other services like rclone or restic servers out of the box, without those endpoints having to implement yet another custom protocol.
but yeah, i'm way too late it seems. figured you might find it interesting anyways... congrats on the performance improvements!
This is not a new protocol, it was added 2 years ago for git-annex's tor hidden service support.
git-annex already supports rclone via a special remote.
This protocol is only used when there's a git-annex repository on the other end, so I don't see any benefit in making it compatible with any other programs. And it has stuff in it that I'm sure if not in the protocol you talked about, like LOCKCONTENT.
While it's true that most keys have a size field giving their size, in the context of this protocol, it's more relevant that the DATA message indicates the number of bytes of content that are going to be transmitted; once the receiver has sucessfully read that many bytes of content, it knows it has the whole content of the key.
When resuming a previous transfer that has been interrupted, if the reciever happened to have all the content of the key, it would send GET with the number of bytes it already has, and the reply would be "DATA 0" which tells it that it already has all the content.
If the whole content of the key has been received and a hash verification fails, git-annex throws away the corrupt data, since this protocol does not provide a way to recover from such a problem.