RecentChanges

design/p2p protocol over http/draft1

todo/git-annex proxies
Changed by: joeyh
Commit type: git
Date: 19:34:58 07/05/24

finalizing HTTP P2p protocol some more
Added v2-v0 endpoints. These are tedious, but will be needed in order to
use the HTTP protocol to proxy to repositories with older git-annex,
where git-annex-shell will be speaking an older version of the protocol.
Changed GET to use 422 when the content is not present. 404 is needed to
detect when a protocol version is not supported.

diff --git a/doc/design/p2p_protocol.mdwn b/doc/design/p2p_protocol.mdwn
index c4f4aac27b..d886376a15 100644
--- a/doc/design/p2p_protocol.mdwn
+++ b/doc/design/p2p_protocol.mdwn
@@ -66,7 +66,7 @@ Now both client and server should use version 1.
 
 ## Cluster cycle prevention
 
-In protocol version 2, immediately after VERSION, the
+In protocol version 2 and above, immediately after VERSION, the
 client can send an additional message that is used to
 prevent cycles when accessing clusters.
 
@@ -135,9 +135,9 @@ The server responds with either SUCCESS or FAILURE.
 
 Note that if the content was not present, SUCCESS will be returned.
 
-In protocol version 2, the server can optionally reply with SUCCESS-PLUS
-or FAILURE-PLUS. Each has a subsequent list of UUIDs of repositories
-that the content was removed from.
+In protocol version 2 and above, the server can optionally reply with
+SUCCESS-PLUS or FAILURE-PLUS. Each has a subsequent list of UUIDs of
+repositories that the content was removed from.
 
 ## Removing content before a specified time
 
diff --git a/doc/design/p2p_protocol_over_http/draft1.mdwn b/doc/design/p2p_protocol_over_http/draft1.mdwn
index 656fc08cee..a776caafe6 100644
--- a/doc/design/p2p_protocol_over_http/draft1.mdwn
+++ b/doc/design/p2p_protocol_over_http/draft1.mdwn
@@ -18,9 +18,7 @@ may want git-annex to use HTTP in eg a LAN.
 ## protocol version
 
 Each request in the protocol is versioned. The versions correspond
-to P2P protocol versions, but for simplicity, the minimum version supported
-over HTTP is version 3. Every implementation of the HTTP protocol must
-support version 3.
+to P2P protocol versions.
 
 The protocol version comes before the request. Eg: `/git-annex/v3/put`
 
@@ -53,6 +51,8 @@ Any request may also optionally include these parameters:
   This parameter can be given multiple times to list several cluster
   gateway UUIDs.
 
+  This parameter is only available for v3 and above.
+
 [Internally, git-annex can use these common parameters, plus the protocol
 version, to create a P2P session. The P2P session is driven through
 the AUTH, VERSION, and BYPASS messages, leaving the session ready to
@@ -67,9 +67,11 @@ It is not part of the P2P protocol per se, but is provided to let
 other clients than git-annex easily download the content of keys from the
 http server.
 
-This behaves the same as `GET /git-annex/v3/key/$key`, although its
+This behaves almost the same as `GET /git-annex/v3/key/$key`, although its
 behavior may change in later versions.
 
+When the key is not present on the server, this returns a 404 Not Found.
+
 ### GET /git-annex/v3/key/$key
 
 Get the content of a key from the server.
@@ -118,7 +120,21 @@ X-git-annex-data-length header indicated, then the data is invalid and
 should not be used. This can happen when eg, the data was being sent from
 an unlocked annexed file, which got modified while it was being sent.
 
-When the content is not present, the server will respond with 404.
+When the content is not present, the server will respond with 
+422 Unprocessable Content.
+
+### GET /git-annex/v2/key/$key
+
+Identical to v3.
+
+### GET /git-annex/v1/key/$key
+
+Identical to v3.
+
+### GET /git-annex/v0/key/$key
+
+Same as v3, except there is no X-git-annex-data-length header.
+Additional checking client-side will be required to validate the data.
 
 ### POST /git-annex/v3/checkpresent
 
@@ -136,6 +152,18 @@ The body of the request is empty.
 The server responds with a JSON object with a "present" field that is true
 if the key is present, or false if it is not present.
 
+### POST /git-annex/v2/checkpresent
+
+Identical to v3.
+
+### POST /git-annex/v1/checkpresent
+
+Identical to v3.
+
+### POST /git-annex/v0/checkpresent
+
+Identical to v3.
+
 ### POST /git-annex/v3/lockcontent
 
 Locks the content of a key on the server, preventing it from being removed.
@@ -160,6 +188,18 @@ If the client disconnects without sending "UNLOCKCONTENT", or the web
 server gets shut down before it can receive that, the content will remain
 locked for at least 10 minutes from when the server sent "SUCCESS".
 
+### POST /git-annex/v2/lockcontent
+
+Identical to v3.
+
+### POST /git-annex/v1/lockcontent
+
+Identical to v3.
+
+### POST /git-annex/v0/lockcontent
+
+Identical to v3.
+
 ### POST /git-annex/v3/remove
 
 Remove a key's content from the server.
@@ -184,6 +224,18 @@ If the server does not allow removing the key due to a policy
 (eg due to being read-only or append-only), it will respond with a JSON
 object with an "error" field that has an error message as its value.
 
+### POST /git-annex/v2/remove
+
+Identical to v3.
+
+### POST /git-annex/v1/remove
+
+Same as v3, except the JSON will not include "plusuuids".
+
+### POST /git-annex/v0/remove
+
+Identival to v1.
+
 ## POST /git-annex/v3/remove-before
 
 Remove a key's content from the server, but only before a specified time.
@@ -269,6 +321,19 @@ If the server does not allow storing the key due eg to a policy
 invalid, or because it ran out of disk space, it will respond with a
 JSON object with an "error" field that has an error message as its value.
 
+### POST /git-annex/v2/put
+
+Identical to v3.
+
+### POST /git-annex/v1/put
+
+Same as v3, except the JSON will not include "plusuuids".
+
+### POST /git-annex/v0/put
+
+Same as v1, except there is no X-git-annex-data-length header.
+Additional checking client-side will be required to validate the data.
+
 ### POST /git-annex/v3/putoffset
 
 Asks the server what `offset` can be used in a `put` of a key.
@@ -289,6 +354,12 @@ The body of the request is empty.
 The server responds with a JSON object with an "offset" field that 
 is the largest allowable offset.
 
+If the server already has the content of the key, it will respond with a
+JSON object with an "alreadyhave" field that is set to true. This JSON
+object may also have a field "plusuuids" that lists 
+the UUIDs of other repositories where the content is stored, in addition to
+the serveruuid.
+
 If the server does not allow storing the key due to a policy
 (eg due to being read-only or append-only), it will respond with a JSON
 object with an "error" field that has an error message as its value.
@@ -299,6 +370,14 @@ part way through a `PUT`, a synthetic empty `DATA` followed by `INVALID`
 will be used to get the P2P protocol back into a state where it will accept
 any request.]
 
+### POST /git-annex/v2/putoffset
+
+Identical to v3.
+
+### POST /git-annex/v1/putoffset
+
+Same as v3, except the JSON will not include "plusuuids".
+
 ## parts of P2P protocol that are not supported over HTTP
 
 `NOTIFYCHANGE` is not supported, but it would be possible to extend
diff --git a/doc/todo/git-annex_proxies.mdwn b/doc/todo/git-annex_proxies.mdwn
index 841fbe13ce..58bb319562 100644
--- a/doc/todo/git-annex_proxies.mdwn
+++ b/doc/todo/git-annex_proxies.mdwn
@@ -28,19 +28,20 @@ Planned schedule of work:
 
 ## work notes
 
-* websockets or something else for LOCKCONTENT over http?

(Diff truncated)

Changed pages:

design/p2p protocol over http/draft1
Changed by: joeyh
Commit type: git
Date: 19:00:05 07/05/24

finalizing HTTP P2P protocol
Managed to avoid netstrings. Actually, using netstrings while streaming
lazy ByteString turns out to be very difficult. So instead, have a
header that specifies the expected amount of data, and then it can just
arrange to send a different amount of data if it needs to indicate
INVALID.
Also improved the interface for GET of a key.

diff --git a/doc/design/p2p_protocol_over_http/draft1.mdwn b/doc/design/p2p_protocol_over_http/draft1.mdwn
index 73a621f404..656fc08cee 100644
--- a/doc/design/p2p_protocol_over_http/draft1.mdwn
+++ b/doc/design/p2p_protocol_over_http/draft1.mdwn
@@ -2,11 +2,6 @@
 
 Draft 1 of a complete [[P2P_protocol]] over HTTP.
 
-## git-annex protocol endpoint and version
-
-The git-annex protocol endpoint is "/git-annex" appended to the HTTP
-url of a git remote.
-
 ## authentication
 
 A git-annex protocol endpoint can optionally operate in readonly mode without
@@ -35,7 +30,8 @@ protocol version.
 
 ## common request parameters
 
-Every request has some common parameters that are always included:
+Every request supports these common parameters, and unless documented
+otherwise, a request requires both of them to be included.
 
 * `clientuuid`  
 
@@ -62,28 +58,69 @@ version, to create a P2P session. The P2P session is driven through
 the AUTH, VERSION, and BYPASS messages, leaving the session ready to
 service requests.]
 
-## binary data framing
+## requests
+
+### GET /git-annex/key/$key
+
+This is a simple, unversioned interface to get a key from the server.
+It is not part of the P2P protocol per se, but is provided to let
+other clients than git-annex easily download the content of keys from the
+http server.
+
+This behaves the same as `GET /git-annex/v3/key/$key`, although its
+behavior may change in later versions.
+
+### GET /git-annex/v3/key/$key
+
+Get the content of a key from the server.
+
+This is designed so it can be used both by a peer in the P2P protocol,
+and by a regular HTTP client that just wants to download a file.
+
+Example:
+
+    > GET /git-annex/v3/key/SHA1--foo&associatedfile=bar&clientuuid=79a5a1f4-07e8-11ef-873d-97f93ca91925&serveruuid=ecf6d4ca-07e8-11ef-8990-9b8c1f696bf6 HTTP/1.1
+    < X-git-annex-data-length: 3
+    < Content-Type: application/octet-stream
+    < 
+    < foo
+
+The key to get is the part of the url after "/git-annex/vN/key/"
+and before any url parameters.
+
+All parameters are optional, including the common parameters, and these:
+
+* `associatedfile`
+
+  The name of a file in the git repository, for informational purposes
+  only.
 
-When a request body or response body includes binary data, eg the content
-of a large file, the body is framed using 
-[netstrings](http://cr.yp.to/proto/netstrings.txt).
+* `offset`
+
+  Number of bytes to skip sending from the beginning of the file. 
 
-The netstring framing is simply the length of the string in ASCII
-digits, followed by the string, and then a comma.
+Request headers are currently ignored, so eg Range requests are
+not supported. (This would be possible to implement, up to a point.)
+
+The body of the request is empty.
+
+The server's response will have a `Content-Type` header of
+`application/octet-stream`.
 
-This allows efficiently sending binary data in one frame, followed by a
-second frame that can contain eg a JSON document.
+The server's response will have a `X-git-annex-data-length` 
+header that indicates the number of bytes of content that are expected to
+be sent. Note that there is no Content-Length header.
 
-For example, a body containing the binary data "foo" followed by
-a JSON document `{"valid": true}` is framed like this:
-    
-    3:foo,15:{"valid": true},
+The body of the response is the content of the key.
 
-## request messages
+If the length of the body is different than what the the
+X-git-annex-data-length header indicated, then the data is invalid and
+should not be used. This can happen when eg, the data was being sent from
+an unlocked annexed file, which got modified while it was being sent.
 
-All the requests below are sent with the HTTP POST method.
+When the content is not present, the server will respond with 404.
 
-### checkpresent
+### POST /git-annex/v3/checkpresent
 
 Checks if a key is currently present on the server.
 
@@ -99,10 +136,17 @@ The body of the request is empty.
 The server responds with a JSON object with a "present" field that is true
 if the key is present, or false if it is not present.
 
-### lockcontent
+### POST /git-annex/v3/lockcontent
 
 Locks the content of a key on the server, preventing it from being removed.
 
+Example:
+
+    > POST /git-annex/v3/lockcontent?key=SHA1--foo&clientuuid=79a5a1f4-07e8-11ef-873d-97f93ca91925&serveruuid=ecf6d4ca-07e8-11ef-8990-9b8c1f696bf6 HTTP/1.1
+    [websocket protocol follows]
+    < SUCCESS
+    > UNLOCKCONTENT
+
 There is one required additional parameter, `key`.
 
 This request opens a websocket between the client and the server.
@@ -116,7 +160,7 @@ If the client disconnects without sending "UNLOCKCONTENT", or the web
 server gets shut down before it can receive that, the content will remain
 locked for at least 10 minutes from when the server sent "SUCCESS".
 
-### remove
+### POST /git-annex/v3/remove
 
 Remove a key's content from the server.
 
@@ -140,7 +184,7 @@ If the server does not allow removing the key due to a policy
 (eg due to being read-only or append-only), it will respond with a JSON
 object with an "error" field that has an error message as its value.
 
-## remove-before
+## POST /git-annex/v3/remove-before
 
 Remove a key's content from the server, but only before a specified time.
 
@@ -158,7 +202,7 @@ removal will fail and the server will respond with: `{"removed": false}`
 This is used to avoid removing content after a point in 
 time where it is no longer locked in other repostitories.
 
-## gettimestamp
+## POST /git-annex/v3/gettimestamp
 
 Gets the current timestamp from the server.
 
@@ -175,7 +219,7 @@ current value of its monotonic clock, as a number of seconds.
 Important: If multiple servers are serving this protocol for the same
 repository, they MUST all use the same monotonic clock.
 
-### put
+### POST /git-annex/v3/put
 
 Store content on the server.
 
@@ -183,8 +227,9 @@ Example:
 
     > POST /git-annex/v3/put?key=SHA1--foo&associatedfile=bar&clientuuid=79a5a1f4-07e8-11ef-873d-97f93ca91925&serveruuid=ecf6d4ca-07e8-11ef-8990-9b8c1f696bf6 HTTP/1.1
     > Content-Type: application/octet-stream
-    > Content-Length: 25
-    > 3:foo,15:{"valid": true},
+    > X-git-annex-object-size: 3
+    > 
+    > foo
     < {"stored": true}
 
 There is one required additional parameter, `key`.
@@ -201,21 +246,16 @@ There are are also these optional parameters:
   Number of bytes that have been omitted from the beginning of the file. 
   Usually this will be determined by making a `putoffset` request.
 
-The body of the request is two items framed with netstrings.
-
-The first item is the content of the key, starting from the specified
-offset or from the beginning when no offset was specified.
-
-The second item is a JSON object.
-
-The JSON object has a field "valid" that is true when the content was not
-changed while it was being sent, or false when modified content was sent
-and should be disregarded by the server. (This corresponds to the `VALID`
-and `INVALID` messages in the P2P protocol.)
-
 The `Content-Type` header should be `application/octet-stream`.
 
-The `Content-Length` header should be set to the length of the body.
+The `X-git-annex-data-length` must be included. It indicates the number
+of bytes of content that are expected to be sent.
+Note that there is no need to send a Content-Length header.

(Diff truncated)

Changed pages:

design/p2p protocol over http/draft1
Changed by: joeyh
Commit type: git
Date: 15:53:03 07/05/24

use netstrings for framing binary data with json at the end
This will be easy to implement with servant. It's also very efficient,
and fairly future-proof. Eg, could add another frame with other data.
This does make it a bit harder to use this protocol, but netstrings
probably take about 5 minutes to implement? Let's see...
import Text.Read
import Data.List
toNetString :: String -> String
toNetString s = show (length s) ++ ":" ++ s ++ ","
nextNetString :: String -> Maybe (String, String)
nextNetString s = case break (== ':') s of
([], _) -> Nothing
(sn, rest) -> do
n <- readMaybe sn
let (v, rest') = splitAt n (drop 1 rest)
return (v, drop 1 rest')
Ok, well, that took about 10 minutes ;-)

diff --git a/doc/design/p2p_protocol_over_http/draft1.mdwn b/doc/design/p2p_protocol_over_http/draft1.mdwn
index 071321537b..73a621f404 100644
--- a/doc/design/p2p_protocol_over_http/draft1.mdwn
+++ b/doc/design/p2p_protocol_over_http/draft1.mdwn
@@ -62,6 +62,23 @@ version, to create a P2P session. The P2P session is driven through
 the AUTH, VERSION, and BYPASS messages, leaving the session ready to
 service requests.]
 
+## binary data framing
+
+When a request body or response body includes binary data, eg the content
+of a large file, the body is framed using 
+[netstrings](http://cr.yp.to/proto/netstrings.txt).
+
+The netstring framing is simply the length of the string in ASCII
+digits, followed by the string, and then a comma.
+
+This allows efficiently sending binary data in one frame, followed by a
+second frame that can contain eg a JSON document.
+
+For example, a body containing the binary data "foo" followed by
+a JSON document `{"valid": true}` is framed like this:
+    
+    3:foo,15:{"valid": true},
+
 ## request messages
 
 All the requests below are sent with the HTTP POST method.
@@ -166,9 +183,8 @@ Example:
 
     > POST /git-annex/v3/put?key=SHA1--foo&associatedfile=bar&clientuuid=79a5a1f4-07e8-11ef-873d-97f93ca91925&serveruuid=ecf6d4ca-07e8-11ef-8990-9b8c1f696bf6 HTTP/1.1
     > Content-Type: application/octet-stream
-    > Content-Length: 20
-    > foo
-    > {"valid": true}
+    > Content-Length: 25
+    > 3:foo,15:{"valid": true},
     < {"stored": true}
 
 There is one required additional parameter, `key`.
@@ -185,14 +201,17 @@ There are are also these optional parameters:
   Number of bytes that have been omitted from the beginning of the file. 
   Usually this will be determined by making a `putoffset` request.
 
-The body of the request is the content of the key, starting from the
-specified offset or from the beginning. After the content of the key,
-there is a newline, followed by a JSON object.
+The body of the request is two items framed with netstrings.
 
-The JSON object has a field "valid" that is true when the content 
-was not changed while it was being sent, or false when modified
-content was sent and should be disregarded by the server. (This corresponds
-to the `VALID` and `INVALID` messages in the P2P protocol.)
+The first item is the content of the key, starting from the specified
+offset or from the beginning when no offset was specified.
+
+The second item is a JSON object.
+
+The JSON object has a field "valid" that is true when the content was not
+changed while it was being sent, or false when modified content was sent
+and should be disregarded by the server. (This corresponds to the `VALID`
+and `INVALID` messages in the P2P protocol.)
 
 The `Content-Type` header should be `application/octet-stream`.
 
@@ -248,8 +267,7 @@ Example:
     > POST /git-annex/v3/get?key=SHA1--foo&associatedfile=bar&clientuuid=79a5a1f4-07e8-11ef-873d-97f93ca91925&serveruuid=ecf6d4ca-07e8-11ef-8990-9b8c1f696bf6 HTTP/1.1
     < Content-Type: application/octet-stream
     > Content-Length: 20
-    > foo
-    > {"valid": true}
+    > 3:foo,15:{"valid": true},
 
 There is one required additional parameter, `key`.
 
@@ -272,9 +290,12 @@ The server's response will have a `Content-Type` header of
 The server's response will have a `Content-Length` header 
 set to the length of the body.
 
-The server's response body is the content of the key, from the specified
-offset. After the content of the key, there is a newline, followed by a
-JSON object.
+The body of the response is two items framed with netstrings.
+
+The first item is the content of the key, starting from the specified
+offset or from the beginning when no offset was specified.
+
+The second item is a JSON object.
 
 The JSON object has a field "valid" that is true when the content 
 was not changed while it was being sent, or false when whatever

Changed pages:

design/p2p protocol over http

design/p2p protocol over http/draft1
Changed by: joeyh
Commit type: git
Date: 14:08:43 07/05/24

thoughts on CGI, and use json

diff --git a/doc/design/p2p_protocol.mdwn b/doc/design/p2p_protocol.mdwn
index 43542c06bb..c4f4aac27b 100644
--- a/doc/design/p2p_protocol.mdwn
+++ b/doc/design/p2p_protocol.mdwn
@@ -133,6 +133,8 @@ To remove a key's content from the server, the client sends:
 
 The server responds with either SUCCESS or FAILURE.
 
+Note that if the content was not present, SUCCESS will be returned.
+
 In protocol version 2, the server can optionally reply with SUCCESS-PLUS
 or FAILURE-PLUS. Each has a subsequent list of UUIDs of repositories
 that the content was removed from.
diff --git a/doc/design/p2p_protocol_over_http.mdwn b/doc/design/p2p_protocol_over_http.mdwn
index c402ca5d8a..6fe7ea4d4e 100644
--- a/doc/design/p2p_protocol_over_http.mdwn
+++ b/doc/design/p2p_protocol_over_http.mdwn
@@ -18,72 +18,80 @@ With the [[passthrough_proxy]], this would let clients configure a single
 http remote that accesses a more complicated network of git-annex
 repositories.
 
-## approach 1: encapsulation
+## integration with git
 
-One approach is to encapsulate the P2P protocol inside HTTP. This has the
-benefit of being simple to think about. It is not very web-native though.
+A webserver that is configured to serve a git repository either serves the
+files in the repository with dumb http, or uses the git-http-backend CGI
+program for url paths under eg `/git/`.
 
-There would be a single API endpoint. The client connects and sends a
-request that encapsulates one or more lines in the P2P protocol. The server
-sends a response that encapsulates one or more lines in the P2P
-protocol.
+To integrate with that, git-annex would need a git-annex-http-backend CGI
+program, that the webserver is configured to run for url paths under
+`/git/.*/annex/`.
 
-For example (eliding the full HTTP responses, only showing the data):
+So, for a remote with an url `http://example.com/git/foo`, git-annex would
+use paths under `http://example.com/git/foo/annex/` to run its CGI.
 
-    > POST /git-annex HTTP/1.0
-    > Content-Type: x-git-annex-p2p
-    > Content-Length: ...
-    > 
-    > AUTH 79a5a1f4-07e8-11ef-873d-97f93ca91925 
-    < AUTH-SUCCESS ecf6d4ca-07e8-11ef-8990-9b8c1f696bf6
+But, the CGI interface is a poor match for the P2P protocol. 
 
-    > POST /git-annex HTTP/1.0
-    > Content-Type: x-git-annex-p2p
-    > Content-Length: ...
-    > 
-    > VERSION 1
-    < VERSION 1
-
-    > POST /git-annex HTTP/1.0
-    > Content-Type: x-git-annex-p2p
-    > Content-Length: ...
-    > 
-    > CHECKPRESENT SHA1--foo
-    < SUCCESS
-
-    > POST /git-annex HTTP/1.0
-    > Content-Type: x-git-annex-p2p
-    > Content-Length: ...
-    > 
-    > PUT bar SHA1--bar
-    < PUT-FROM 0
+A particular problem is that `LOCKCONTENT` would need to be in one CGI
+request, followed by another request to `UNLOCKCONTENT`. Unless
+git-annex-http-backend forked a daemon to keep the content locked, it would
+not be able to retain a file lock across the 2 requests. While the 10
+minute retention lock would paper over that, UNLOCKCONTENT would not be
+able to delete the retention lock, because there is no way to know if
+another LOCKCONTENT was received later. So LOCKCONTENT would always lock
+content for 10 minutes. Which would result in some undesirable behaviors.
 
-    > POST /git-annex HTTP/1.0
-    > Content-Type: x-git-annex-p2p
-    > Content-Length: ...
-    > 
-    > DATA 3
-    > foo
-    > VALID
-    < SUCCESS
+Another problem is with proxies and clusters. The CGI would need to open
+ssh (or http) connections to the proxied repositories and cluster nodes
+each time it is run. That would add a lot of latency to every request.
+
+And running a git-annex process once per CGI request also makes git-annex's
+own startup speed, which is ok but not great, add latency. And each time
+the CGI needed to change the git-annex branch, it would have to commit on
+shutdown. Lots of time and space optimisations would be prevented by using
+the CGI interface.
+
+So, rather than having the CGI program do anything in the repository
+itself, have it pass each request through to a long-running server.
+(This does have the downside that files would get double-copied
+through the CGI, which adds some overhead.)
+A reasonable way to do that would be to have a webserver speaking a
+HTTP version of the git-annex P2P protocol and the CGI just talks to that.
 
-Note that, since VERSION is negotiated in one request, the HTTP server
-needs to know that a series of requests are part of the same P2P protocol
-session. In the example above, it would not have a good way to do that.
-One solution would be to add a session identifier UUID to each request.
+The CGI program then becomes tiny, and just needs to know the url to
+connect to the git-annex HTTP server.
 
-## approach 2: websockets
+Alternatively, a remote's configuration could include that url, and
+then we don't need the complication and overhead of the CGI program at all.
+Eg:
+
+    git config remote.origin.annex-url http://example.com:8080/
+
+So, the rest of this design will focus on implementing that. The CGI
+program can be added later if desired, so avoid users needing to configure
+an additional thing.
+
+Note that, one nice benefit of having a separate annex-url is it allows
+having remote.origin.url on eg github, but with an annex-url configured
+that remote can also be used as a git-annex repository.
+
+## approach 1: websockets
 
 The client connects to the server over a websocket. From there on,
 the protocol is encapsulated in websockets.
 
-This seems nice and simple, but again not very web native. 
+This seems nice and simple to implement, but not very web native. Anyone
+wanting to talk to this web server would need to understand the P2P
+protocol. Just to upload a file would need to deal with AUTH,
+AUTH-SUCCESS, AUTH-FAILURE, VERSION, PUT, ALREADY-HAVE, PUT-FROM, DATA,
+INVALID, VALID, SUCCESS, and FAILURE messages. Seems like a lot.
 
-Some requests like `LOCKCONTENT` seem likely to need full duplex
-communication like websockets provide. But, it might be more web native to
-only use websockets for that request, and not for everything.
+Some requests like `LOCKCONTENT` do need full duplex communication like
+websockets provide. But, it might be more web native to only use websockets
+for that request, and not for everything.
 
-## approach 3: HTTP API
+## approach 2: web-native API
 
 Another approach is to define a web-native API with endpoints that
 correspond to each action in the P2P protocol. 
@@ -101,13 +109,13 @@ Something like this:
 
     > POST /git-annex/v1/PUT?key=SHA1--foo&associatedfile=bar&put-from=0&clientuuid=79a5a1f4-07e8-11ef-873d-97f93ca91925&serveruuid=ecf6d4ca-07e8-11ef-8990-9b8c1f696bf6 HTTP/1.0
     > Content-Type: application/octet-stream
-    > Content-Length: 4
-    > foo1
-    < SUCCESS
+    > Content-Length: 20
+    > foo
+    > {"valid": true}
+    < {"stored": true}
 
-(In the last example above "foo" is the content, there is an additional byte at the end that
-is 1 for VALID and 0 for INVALID. This seems better than needing an entire
-other request to indicate validitity.)
+(In the last example above "foo" is the content, it is followed by a line of json.
+This seems better than needing an entire other request to indicate validitity.)
 
 This needs a more complex spec. But it's easier for others to implement,
 especially since it does not need a session identifier, so the HTTP server can 
diff --git a/doc/design/p2p_protocol_over_http/draft1.mdwn b/doc/design/p2p_protocol_over_http/draft1.mdwn
index 20ec846514..071321537b 100644
--- a/doc/design/p2p_protocol_over_http/draft1.mdwn
+++ b/doc/design/p2p_protocol_over_http/draft1.mdwn
@@ -73,14 +73,14 @@ Checks if a key is currently present on the server.
 Example:
 
     > POST /git-annex/v3/checkpresent?key=SHA1--foo&clientuuid=79a5a1f4-07e8-11ef-873d-97f93ca91925&serveruuid=ecf6d4ca-07e8-11ef-8990-9b8c1f696bf6 HTTP/1.1
-    < SUCCESS
+    < {"present": true}
 
 There is one required additional parameter, `key`.
 
 The body of the request is empty.
 
-The server responds with "SUCCESS" if the key is present
-or "FAILURE" if it is not present.
+The server responds with a JSON object with a "present" field that is true
+if the key is present, or false if it is not present.
 
 ### lockcontent
 
@@ -106,24 +106,22 @@ Remove a key's content from the server.
 Example:
 
     > POST /git-annex/v3/remove?key=SHA1--foo&clientuuid=79a5a1f4-07e8-11ef-873d-97f93ca91925&serveruuid=ecf6d4ca-07e8-11ef-8990-9b8c1f696bf6 HTTP/1.1
-    < SUCCESS
+    < {"removed": true}
 
 There is one required additional parameter, `key`.

(Diff truncated)

Changed pages:

forum/Questions about borg special remotes/comment 3 a036be41663d81a79fe47e9021208169
Changed by: git-annex
Commit type: web
Date: 10:22:46 07/05/24

Added a comment

diff --git a/doc/forum/Questions_about_borg_special_remotes/comment_3_a036be41663d81a79fe47e9021208169._comment b/doc/forum/Questions_about_borg_special_remotes/comment_3_a036be41663d81a79fe47e9021208169._comment
new file mode 100644
index 0000000000..1a7b653b09
--- /dev/null
+++ b/doc/forum/Questions_about_borg_special_remotes/comment_3_a036be41663d81a79fe47e9021208169._comment
@@ -0,0 +1,9 @@
+[[!comment format=mdwn
+ username="git-annex@4a0625db6ced1ac00744697d5bac41393bcde646"
+ nickname="git-annex"
+ avatar="http://cdn.libravatar.org/avatar/d9fd8db33996d9a1da6b31045c43accf"
+ subject="comment 3"
+ date="2024-07-05T10:22:45Z"
+ content="""
+Is there a way to configure a certain BORG_PASSCOMMAND to always be used for a given remote. In general, is there a way to set environment variables per remote?
+"""]]

Changed pages:

design/p2p protocol over http/draft1
Changed by: joeyh
Commit type: git
Date: 19:26:05 07/04/24

update

diff --git a/doc/design/p2p_protocol.mdwn b/doc/design/p2p_protocol.mdwn
index dd3800f48b..43542c06bb 100644
--- a/doc/design/p2p_protocol.mdwn
+++ b/doc/design/p2p_protocol.mdwn
@@ -115,13 +115,16 @@ the client sends:
 
 The server responds with either SUCCESS or FAILURE.
 The former indicates the content is locked. It will remain
-locked until 10 minutes after the connection is broken, or until
-the client sends:
+locked until the client sends:
 
 	UNLOCKCONTENT Key
 
 The server makes no response to that.
 
+If the connection is broken before the client sends UNLOCKCONTENT,
+the content will remain locked for at least 10 minutes from when the server
+sent SUCCESS.
+
 ## Removing content
 
 To remove a key's content from the server, the client sends:
diff --git a/doc/design/p2p_protocol_over_http/draft1.mdwn b/doc/design/p2p_protocol_over_http/draft1.mdwn
index 261a400953..20ec846514 100644
--- a/doc/design/p2p_protocol_over_http/draft1.mdwn
+++ b/doc/design/p2p_protocol_over_http/draft1.mdwn
@@ -92,31 +92,12 @@ This request opens a websocket between the client and the server.
 The server sends "SUCCESS" over the websocket once it has locked
 the content. Or it sends "FAILURE" if it is unable to lock the content.
 
-Once the server has sent "SUCCESS", the content remains locked as long as
-the client remains connected to the websocket. When the client disconnects,
-or closes the websocket, the server unlocks the content.
+Once the server has sent "SUCCESS", the content remains locked 
+until the client sends "UNLOCKCONTENT" over the websocket.
 
-XXX What happens if the connection times out? Will the client notice that
-in time? How does this work with P2P over ssh?
-
-### limit-remove
-
-Limit the next requested removal of a key to occur within a specified
-number of seconds.
-
-Example:
-
-    > POST /git-annex/v3/limit-remove?seconds=600&key=SHA1--foo&clientuuid=79a5a1f4-07e8-11ef-873d-97f93ca91925&serveruuid=ecf6d4ca-07e8-11ef-8990-9b8c1f696bf6 HTTP/1.1
-    < SUCCESS
-
-There are two required additional parameters, `key` and `seconds`.
-
-The body of the request is empty.
-
-The server responds with "SUCCESS".
-
-The server will check the next `remove` request, and if it's for the same key,
-and more time has elapsed, it will refuse to remove the key's content.
+If the client disconnects without sending "UNLOCKCONTENT", or the web
+server gets shut down before it can receive that, the content will remain
+locked for at least 10 minutes from when the server sent "SUCCESS".
 
 ### remove
 
@@ -156,9 +137,9 @@ Example:
 This is the same as the `remove` request, but with an additional parameter,
 `timestamp`.
 
-If the server's clock is past the specified timestamp, the removal will
-fail. This is used to avoid removing content after a point in time where it
-is no longer locked in other repostitories.
+If the server's monotonic clock is past the specified timestamp, the
+removal will fail. This is used to avoid removing content after a point in
+time where it is no longer locked in other repostitories.
 
 ## gettimestamp

Changed pages:

todo/git-annex proxies
Changed by: joeyh
Commit type: git
Date: 19:18:06 07/04/24

update

diff --git a/doc/todo/git-annex_proxies.mdwn b/doc/todo/git-annex_proxies.mdwn
index 4c33d87abb..841fbe13ce 100644
--- a/doc/todo/git-annex_proxies.mdwn
+++ b/doc/todo/git-annex_proxies.mdwn
@@ -28,18 +28,8 @@ Planned schedule of work:
 
 ## work notes
 
-* [[todo/P2P_locking_connection_drop_safety]] is blocking http protocol,
-  because it will involve protocol changes and we need to get locking
-  right in the http protocol from the beginning.
-
-  Status: Content retention files implemented. P2P LOCKCONTENT uses a 10
-  minute retention in case it gets killed. P2P protocol changes started.
-
 * websockets or something else for LOCKCONTENT over http?
 
-* will client notice promptly when http connection with server
-  is closed in LOCKCONTENT?
-
 * Should integrate into an existing web server similarly to git
   smart http server. May as well also support it as a standalone
   webserver.
@@ -47,6 +37,10 @@ Planned schedule of work:
 * endpoint versioning should include v1 and v0 to support connections
   proxied from older clients
 
+## completed items for July's work on p2p protocol over http
+
+* addressed [[doc/todo/P2P_locking_connection_drop_safety]]
+
 ## items deferred until later for [[design/passthrough_proxy]]
 
 * Check annex.diskreserve when proxying for special remotes

Changed pages:

todo/P2P locking connection drop safety
Changed by: joeyh
Commit type: git
Date: 19:08:33 07/04/24

REMOVE-BEFORE and GETTIMESTAMP proxying
For clusters, the timestamps have to be translated, since each node can
have its own idea about what time it is. To translate a timestamp, the
proxy remembers what time it asked the node for a timestamp in
GETTIMESTAMP, and applies the delta as an offset in REMOVE-BEFORE.
This does mean that a remove from a cluster has to call GETTIMESTAMP on
every node before dropping from nodes. Not very efficient. Although
currently it tries to drop from every single node anyway, which is also
not very efficient.
I thought about caching the GETTIMESTAMP from the nodes on the first
call. That would improve efficiency. But, since monotonic clocks on
!Linux don't advance when the computer is suspended, consider what might
happen if one node was suspended for a while, then came back. Its
monotonic timestamp would end up behind where the proxying expects it to
be. Would that result in removing when it shouldn't, or refusing to
remove when it should? Have not thought it through. Either way, a
cluster behaving strangly for an extended period of time because one
of its nodes was briefly asleep doesn't seem like good behavior.

diff --git a/Annex/Cluster.hs b/Annex/Cluster.hs
index 3c0bc7e3a2..7f20e1486f 100644
--- a/Annex/Cluster.hs
+++ b/Annex/Cluster.hs
@@ -67,7 +67,8 @@ proxyCluster clusteruuid proxydone servermode clientside protoerrhandler = do
 		(selectnode, closenodes) <- clusterProxySelector clusteruuid
 			protocolversion bypassuuids
 		concurrencyconfig <- getConcurrencyConfig
-		proxy proxydone proxymethods servermode clientside 
+		proxystate <- liftIO mkProxyState
+		proxy proxydone proxymethods proxystate servermode clientside 
 			(fromClusterUUID clusteruuid)
 			selectnode concurrencyconfig protocolversion
 			othermsg (protoerrhandler closenodes)
@@ -107,6 +108,7 @@ clusterProxySelector clusteruuid protocolversion (Bypass bypass) = do
 		-- could be out of date, actually try to remove from every
 		-- node.
 		, proxyREMOVE = const (pure nodes)
+		, proxyGETTIMESTAMP = pure nodes
 		-- Content is not locked on the cluster as a whole,
 		-- instead it can be locked on individual nodes that are
 		-- proxied to the client.
diff --git a/Command/P2PStdIO.hs b/Command/P2PStdIO.hs
index 910d9cb7cc..5563758c8d 100644
--- a/Command/P2PStdIO.hs
+++ b/Command/P2PStdIO.hs
@@ -76,7 +76,9 @@ performProxy clientuuid servermode r = do
 			closeRemoteSide remoteside
 			p2pDone
 		let errhandler = p2pErrHandler (closeRemoteSide remoteside)
-		let runproxy othermsg' = proxy closer proxymethods
+		proxystate <- liftIO mkProxyState
+		let runproxy othermsg' = proxy closer
+			proxymethods proxystate
 			servermode clientside
 			(Remote.uuid r)
 			(singleProxySelector remoteside)
diff --git a/P2P/Proxy.hs b/P2P/Proxy.hs
index 53cc4ff947..32ef7166fe 100644
--- a/P2P/Proxy.hs
+++ b/P2P/Proxy.hs
@@ -15,6 +15,7 @@ import qualified Annex
 import P2P.Protocol
 import P2P.IO
 import Utility.Metered
+import Utility.MonotonicClock
 import Git.FilePath
 import Types.Concurrency
 import Annex.Concurrent
@@ -26,16 +27,22 @@ import Control.Concurrent.Async
 import qualified Control.Concurrent.MSem as MSem
 import qualified Data.ByteString.Lazy as L
 import qualified Data.Set as S
+import qualified Data.Map as M
+import Data.Unique
 import GHC.Conc
 
 type ProtoCloser = Annex ()
 
 data ClientSide = ClientSide RunState P2PConnection
 
+newtype RemoteSideId = RemoteSideId Unique
+	deriving (Eq, Ord)
+
 data RemoteSide = RemoteSide
 	{ remote :: Remote
 	, remoteConnect :: Annex (Maybe (RunState, P2PConnection, ProtoCloser))
 	, remoteTMVar :: TMVar (RunState, P2PConnection, ProtoCloser)
+	, remoteSideId :: RemoteSideId
 	}
 
 mkRemoteSide :: Remote -> Annex (Maybe (RunState, P2PConnection, ProtoCloser)) -> Annex RemoteSide
@@ -43,6 +50,7 @@ mkRemoteSide r remoteconnect = RemoteSide
 	<$> pure r
 	<*> pure remoteconnect
 	<*> liftIO (atomically newEmptyTMVar)
+	<*> liftIO (RemoteSideId <$> newUnique)
 
 runRemoteSide :: RemoteSide -> Proto a -> Annex (Either ProtoFailure a)
 runRemoteSide remoteside a = 
@@ -71,6 +79,9 @@ data ProxySelector = ProxySelector
 	, proxyUNLOCKCONTENT :: Annex (Maybe RemoteSide)
 	, proxyREMOVE :: Key -> Annex [RemoteSide]
 	-- ^ remove from all of these remotes
+	, proxyGETTIMESTAMP :: Annex [RemoteSide]
+	-- ^ should send every remote that proxyREMOVE can
+	-- ever return for any key
 	, proxyGET :: Key -> Annex (Maybe RemoteSide)
 	, proxyPUT :: AssociatedFile -> Key -> Annex [RemoteSide]
 	-- ^ put to some/all of these remotes
@@ -82,6 +93,7 @@ singleProxySelector r = ProxySelector
 	, proxyLOCKCONTENT = const (pure (Just r))
 	, proxyUNLOCKCONTENT = pure (Just r)
 	, proxyREMOVE = const (pure [r])
+	, proxyGETTIMESTAMP = pure [r]
 	, proxyGET = const (pure (Just r))
 	, proxyPUT = const (const (pure [r]))
 	}
@@ -178,12 +190,23 @@ getClientBypass _ _ (Just othermsg) cont _ =
 	-- Pass along non-BYPASS message from version 0 client.
 	cont (Bypass S.empty, (Just othermsg))
 
+data ProxyState = ProxyState
+	{ proxyRemoteLatestTimestamps :: TVar (M.Map RemoteSideId MonotonicTimestamp)
+	, proxyRemoteLatestLocalTimestamp :: TVar (Maybe MonotonicTimestamp)
+	}
+
+mkProxyState :: IO ProxyState
+mkProxyState = ProxyState
+	<$> newTVarIO mempty
+	<*> newTVarIO Nothing
+
 {- Proxy between the client and the remote. This picks up after
  - sendClientProtocolVersion.
  -}
 proxy 
 	:: Annex r
 	-> ProxyMethods
+	-> ProxyState
 	-> ServerMode
 	-> ClientSide
 	-> UUID
@@ -197,7 +220,7 @@ proxy
 	-- ^ non-VERSION message that was received from the client when
 	-- negotiating protocol version, and has not been responded to yet
 	-> ProtoErrorHandled r
-proxy proxydone proxymethods servermode (ClientSide clientrunst clientconn) remoteuuid proxyselector concurrencyconfig (ProtocolVersion protocolversion) othermsg protoerrhandler = do
+proxy proxydone proxymethods proxystate servermode (ClientSide clientrunst clientconn) remoteuuid proxyselector concurrencyconfig (ProtocolVersion protocolversion) othermsg protoerrhandler = do
 	case othermsg of
 		Nothing -> proxynextclientmessage ()
 		Just message -> proxyclientmessage (Just message)
@@ -238,6 +261,13 @@ proxy proxydone proxymethods servermode (ClientSide clientrunst clientconn) remo
 			remotesides <- proxyREMOVE proxyselector k
 			servermodechecker checkREMOVEServerMode $
 				handleREMOVE remotesides k message
+		REMOVE_BEFORE _ k -> do
+			remotesides <- proxyREMOVE proxyselector k
+			servermodechecker checkREMOVEServerMode $
+				handleREMOVE remotesides k message
+		GETTIMESTAMP -> do
+			remotesides <- proxyGETTIMESTAMP proxyselector
+			handleGETTIMESTAMP remotesides
 		GET _ _ k -> proxyGET proxyselector k >>= \case
 			Just remoteside -> handleGET remoteside message
 			Nothing -> 
@@ -278,6 +308,7 @@ proxy proxydone proxymethods servermode (ClientSide clientrunst clientconn) remo
 		PUT_FROM _ -> protoerr
 		ALREADY_HAVE -> protoerr
 		ALREADY_HAVE_PLUS _ -> protoerr
+		TIMESTAMP _ -> protoerr
 		-- Early messages that the client should not send now.
 		AUTH _ _ -> protoerr
 		VERSION _ -> protoerr
@@ -318,15 +349,70 @@ proxy proxydone proxymethods servermode (ClientSide clientrunst clientconn) remo
 		_ <- client $ net $ sendMessage (ERROR "protocol error X")
 		giveup "protocol error M"
 	
+	-- When there is a single remote, reply with its timestamp,
+	-- to avoid needing timestamp translation.
+	handleGETTIMESTAMP (remoteside:[]) = do
+		liftIO $ hPutStrLn stderr "!!!! single remote side"
+		liftIO $ atomically $ do
+			writeTVar (proxyRemoteLatestTimestamps proxystate)
+				mempty
+			writeTVar (proxyRemoteLatestLocalTimestamp proxystate)
+				Nothing
+		proxyresponse remoteside GETTIMESTAMP
+			(const proxynextclientmessage)
+	-- When there are multiple remotes, reply with our local timestamp,
+	-- and do timestamp translation when sending REMOVE-FROM.
+	handleGETTIMESTAMP remotesides = do
+		-- Order of getting timestamps matters.
+		-- Getting the local time after the time of the remotes
+		-- means that if there is some delay in getting the time
+		-- from a remote, that is reflected in the local time,
+		-- and so reduces the allowed time.
+		remotetimes <- (M.fromList . mapMaybe join) <$> getremotetimes 
+	 	localtime <- liftIO currentMonotonicTimestamp
+		liftIO $ atomically $ do
+			writeTVar (proxyRemoteLatestTimestamps proxystate)
+				remotetimes
+			writeTVar (proxyRemoteLatestLocalTimestamp proxystate)
+				(Just localtime)
+		protoerrhandler proxynextclientmessage $
+			client $ net $ sendMessage (TIMESTAMP localtime)
+	  where
+		getremotetimes = forMC concurrencyconfig remotesides $ \r ->
+			runRemoteSideOrSkipFailed r $ do
+				net $ sendMessage GETTIMESTAMP
+				net receiveMessage >>= return . \case
+					Just (TIMESTAMP ts) ->
+						Just (remoteSideId r, ts)
+					_ -> Nothing
+
+	proxyTimestamp ts _ _ Nothing = ts -- not proxying timestamps
+	proxyTimestamp ts r tsm (Just correspondinglocaltime) =
+		case M.lookup (remoteSideId r) tsm of
+			Just oldts -> oldts + (ts - correspondinglocaltime)
+			Nothing -> ts -- not reached
+

(Diff truncated)

Changed pages:

todo/P2P locking connection drop safety
Changed by: joeyh
Commit type: git
Date: 17:42:09 07/04/24

use REMOVE-BEFORE in P2P protocol
Only clusters still need to be fixed to close this todo.

diff --git a/Annex/SafeDropProof.hs b/Annex/SafeDropProof.hs
index b54b627cbc..eca3fa6f51 100644
--- a/Annex/SafeDropProof.hs
+++ b/Annex/SafeDropProof.hs
@@ -23,12 +23,12 @@ safeDropProofExpired :: Annex ()
 safeDropProofExpired = do
 	showNote "unsafe"
 	showLongNote $ UnquotedString
-		"Dropping took too long, and locks on remotes may have expired."
+		"Dropping took too long, and locks may have expired."
 
 checkSafeDropProofEndTime :: Maybe SafeDropProof -> IO Bool
 checkSafeDropProofEndTime p = case safeDropProofEndTime =<< p of
 	Nothing -> return True
-	Just t -> do
+	Just endtime -> do
 		now <- getPOSIXTime
-		return (t < now)
+		return (endtime > now)
 
diff --git a/P2P/Annex.hs b/P2P/Annex.hs
index 44e060ca20..8d7348e36f 100644
--- a/P2P/Annex.hs
+++ b/P2P/Annex.hs
@@ -28,6 +28,7 @@ import Annex.Verify
 
 import Control.Monad.Free
 import Control.Concurrent.STM
+import Data.Time.Clock.POSIX
 import qualified Data.ByteString as S
 
 -- Full interpreter for Proto, that can receive and send objects.
@@ -156,7 +157,10 @@ runLocal runst runner a = case a of
 	UpdateMeterTotalSize m sz next -> do
 		liftIO $ setMeterTotalSize m sz
 		runner next
-	RunValidityCheck checkaction next -> runner . next =<< checkaction
+	RunValidityCheck checkaction next ->
+		runner . next =<< checkaction
+	GetLocalCurrentTime next ->
+		runner . next =<< liftIO getPOSIXTime
   where
 	transfer mk k af sd ta = case runst of
 		-- Update transfer logs when serving.
diff --git a/P2P/Protocol.hs b/P2P/Protocol.hs
index 5aad27920a..3f3d4a4dbd 100644
--- a/P2P/Protocol.hs
+++ b/P2P/Protocol.hs
@@ -42,6 +42,7 @@ import qualified Data.ByteString as B
 import qualified Data.ByteString.Lazy as L
 import qualified Data.Set as S
 import Data.Char
+import Data.Time.Clock.POSIX
 import Control.Applicative
 import Prelude
 
@@ -327,6 +328,8 @@ data LocalF c
 	-- not known until the data is being received.
 	| RunValidityCheck (Annex Validity) (Validity -> c)
 	-- ^ Runs a deferred validity check.
+	| GetLocalCurrentTime (POSIXTime -> c)
+	-- ^ Gets the local time.
 	deriving (Functor)
 
 type Local = Free LocalF
@@ -397,9 +400,49 @@ lockContentWhile runproto key a = bracket setup cleanup a
 	cleanup False = return ()
 
 remove :: Maybe SafeDropProof -> Key -> Proto (Either String Bool, Maybe [UUID])
-remove proof key = do
-	net $ sendMessage (REMOVE key)
-	checkSuccessFailurePlus
+remove proof key = 
+	case safeDropProofEndTime =<< proof of
+		Nothing -> removeanytime
+		Just endtime -> do
+			ver <- net getProtocolVersion
+			if ver >= ProtocolVersion 3
+				then removeBefore endtime key
+				-- Peer is too old to support REMOVE-BEFORE
+				else removeanytime
+  where
+	removeanytime = do
+		net $ sendMessage (REMOVE key)
+		checkSuccessFailurePlus
+
+{- The endtime is the last local time at which the key can be removed.
+ - To tell the remote how long it has to remove the key, get its current
+ - timestamp, and add to it the number of seconds from the current local
+ - time until the endtime.
+ -
+ - Order of retrieving timestamps matters. Getting the local time after the
+ - remote timestamp means that, if there is some delay in getting the
+ - response from the remote, that is reflected in the local time, and so
+ - reduces the allowed time.
+ -}
+removeBefore :: POSIXTime -> Key -> Proto (Either String Bool, Maybe [UUID])
+removeBefore endtime key = do
+	net $ sendMessage GETTIMESTAMP
+	net receiveMessage >>= \case
+		Just (TIMESTAMP remotetime) -> do
+			localtime <- local getLocalCurrentTime
+			let timeleft = endtime - localtime
+			let timeleft' = MonotonicTimestamp (floor timeleft)
+			let remoteendtime = remotetime + timeleft'
+			if timeleft <= 0
+				then return (Right False, Nothing)
+				else do
+					net $ sendMessage $
+						REMOVE_BEFORE remoteendtime key
+					checkSuccessFailurePlus	
+		Just (ERROR err) -> return (Left err, Nothing)
+		_ -> do
+			net $ sendMessage (ERROR "expected TIMESTAMP")
+			return (Right False, Nothing)
 
 get :: FilePath -> Key -> Maybe IncrementalVerifier -> AssociatedFile -> Meter -> MeterUpdate -> Proto (Bool, Verification)
 get dest key iv af m p = 
diff --git a/Utility/MonotonicClock.hs b/Utility/MonotonicClock.hs
index 1af10187e3..d7d2ada7e5 100644
--- a/Utility/MonotonicClock.hs
+++ b/Utility/MonotonicClock.hs
@@ -5,6 +5,7 @@
  - License: BSD-2-clause
  -}
 
+{-# LANGUAGE GeneralizedNewtypeDeriving #-}
 {-# LANGUAGE CPP #-}
 
 module Utility.MonotonicClock where
@@ -19,7 +20,7 @@ import Utility.Exception
 #endif
 
 newtype MonotonicTimestamp = MonotonicTimestamp Integer
-	deriving (Show, Eq, Ord)
+	deriving (Show, Eq, Ord, Num)
 
 -- On linux, this uses a clock that advances while the system is suspended,
 -- except for on very old kernels (eg 2.6.32).
diff --git a/doc/todo/P2P_locking_connection_drop_safety.mdwn b/doc/todo/P2P_locking_connection_drop_safety.mdwn
index a3428fa613..18ae8015a4 100644
--- a/doc/todo/P2P_locking_connection_drop_safety.mdwn
+++ b/doc/todo/P2P_locking_connection_drop_safety.mdwn
@@ -47,7 +47,7 @@ remotedaemon` for tor, or something similar for future P2P over HTTP
 process is kept running. An admin may bounce the HTTP server at any point,
 or the whole system reboot.
 
-----
+## retention locking
 
 So, this needs a way to make lockContentShared guarentee it remains
 locked for an amount of time even after the process has exited.
@@ -64,41 +64,7 @@ OTOH putting the timestamp in the lock file may be hard (eg on Windows).
 > P2P LOCKCONTENT uses a 10 minute retention in case it gets killed,
 > but other values can be used in the future safely.
 
-----
-
-Extending the P2P protocol is a bit tricky, because the same P2P
-protocol connection could be used for several different things at
-the same time. A PRE-REMOVE N Key might be followed by removals of other
-keys, and eventually a removal of the requested key. There are
-sometimes pools of P2P connections that get used like this.
-So the server would need to cache some number of PRE-REMOVE timestamps.
-How many?
-
-Certainly care would need to be taken to send PRE-REMOVE to the same
-connection as REMOVE. How?
-
-Could this be done without extending the REMOVE side of the P2P protocol?
-
-1. check start time
-2. LOCKCONTENT
-3. prepare to remove
-4. in checkVerifiedCopy, 
-   check current time.. fail if more than 10 minutes from start
-5. REMOVE
-
-The issue with this is that git-annex could be paused for any amount of
-time between steps 4 and 5. Usually it won't pause.. 
-mkSafeDropProof calls checkVerifiedCopy and constructs the proof,
-and then it immediately sends REMOVE. But of course sending REMOVE
-could take arbitrarily long. Or git-annex could be paused at just the wrong
-point.
-
-Ok, let's reconsider... Add GETTIMESTAMP which causes the server to
-return its current timestamp. The same timestamp must be returned on any
-connection to the server, eg the server must have a single clock.
-That can be called before LOCKCONTENT.
-Then REMOVE Key Timestamp can fail if the current time is past the
-specified timestamp. 
+## clusters
 
 How to handle this when proxying to a cluster? In a cluster, each node
 has a different clock. So GETTIMESTAMP will return a bunch of times.
@@ -107,13 +73,24 @@ Then REMOVE Key Timestamp can have the timestamp adjusted when it's sent
 out to each client, by calling GETTIMESTAMP again and applying the offsets
 between the cluster's clock and each node's clock.
 
-This approach would need to use a monotonic clock!

(Diff truncated)

Changed pages:

todo/P2P locking connection drop safety
Changed by: joeyh
Commit type: git
Date: 16:23:46 07/04/24

toward SafeDropProof expiry checking
Added Maybe POSIXTime to SafeDropProof, which gets set when the proof is
based on a LockedCopy. If there are several LockedCopies, it uses the
closest expiry time. That is not optimal, it may be that the proof
expires based on one LockedCopy but another one has not expired. But
that seems unlikely to really happen, and anyway the user can just
re-run a drop if it fails due to expiry.
Pass the SafeDropProof to removeKey, which is responsible for checking
it for expiry in situations where that could be a problem. Which really
only means in Remote.Git.
Made Remote.Git check expiry when dropping from a local remote.
Checking expiry when dropping from a P2P remote is not yet implemented.
P2P.Protocol.remove has SafeDropProof plumbed through to it for that
purpose.
Fixing the remaining 2 build warnings should complete this work.
Note that the use of a POSIXTime here means that if the clock gets set
forward while git-annex is in the middle of a drop, it may say that
dropping took too long. That seems ok. Less ok is that if the clock gets
turned back a sufficient amount (eg 5 minutes), proof expiry won't be
noticed. It might be better to use the Monotonic clock, but that doesn't
advance when a laptop is suspended, and while there is the linux
Boottime clock, that is not available on other systems. Perhaps a
combination of POSIXTime and the Monotonic clock could detect laptop
suspension and also detect clock being turned back?
There is a potential future flag day where
p2pDefaultLockContentRetentionDuration is not assumed, but is probed
using the P2P protocol, and peers that don't support it can no longer
produce a LockedCopy. Until that happens, when git-annex is
communicating with older peers there is a risk of data loss when
a ssh connection closes during LOCKCONTENT.

diff --git a/Annex/Content.hs b/Annex/Content.hs
index 2a52b59400..784fbbf1da 100644
--- a/Annex/Content.hs
+++ b/Annex/Content.hs
@@ -141,7 +141,7 @@ lockContentShared key mduration a = do
 		ifM (inAnnex key)
 			( do
 				u <- getUUID
-				withVerifiedCopy LockedCopy u (return True) a
+				withVerifiedCopy LockedCopy u (return (Right True)) a
 			, notpresent
 			)
   where
diff --git a/Annex/NumCopies.hs b/Annex/NumCopies.hs
index c4722c751d..6ec339cae8 100644
--- a/Annex/NumCopies.hs
+++ b/Annex/NumCopies.hs
@@ -29,6 +29,7 @@ module Annex.NumCopies (
 
 import Annex.Common
 import qualified Annex
+import Annex.SafeDropProof
 import Types.NumCopies
 import Logs.NumCopies
 import Logs.Trust
@@ -227,6 +228,10 @@ data UnVerifiedCopy = UnVerifiedRemote Remote | UnVerifiedHere
 {- Verifies that enough copies of a key exist among the listed remotes,
  - to safely drop it, running an action with a proof if so, and
  - printing an informative message if not.
+ -
+ - Note that the proof is checked to still be valid at the current time
+ - before running the action, but when dropping the key may take some time,
+ - the proof's time may need to be checked again.
  -}
 verifyEnoughCopiesToDrop
 	:: String -- message to print when there are no known locations
@@ -246,14 +251,14 @@ verifyEnoughCopiesToDrop nolocmsg key dropfrom removallock neednum needmin skip
   where
 	helper bad missing have [] lockunsupported =
 		liftIO (mkSafeDropProof neednum needmin have removallock) >>= \case
-			Right proof -> dropaction proof
+			Right proof -> checkprooftime proof
 			Left stillhave -> do
 				notEnoughCopies key dropfrom neednum needmin stillhave (skip++missing) bad nolocmsg lockunsupported
 				nodropaction
 	helper bad missing have (c:cs) lockunsupported
 		| isSafeDrop neednum needmin have removallock =
 			liftIO (mkSafeDropProof neednum needmin have removallock) >>= \case
-				Right proof -> dropaction proof
+				Right proof -> checkprooftime proof
 				Left stillhave -> helper bad missing stillhave (c:cs) lockunsupported
 		| otherwise = case c of
 			UnVerifiedHere -> lockContentShared key Nothing contverified
@@ -294,6 +299,14 @@ verifyEnoughCopiesToDrop nolocmsg key dropfrom removallock neednum needmin skip
 				, MC.Handler (\ (_e :: SomeException) -> fallback)
 				]
 		Nothing -> fallback
+	
+	checkprooftime proof = 
+		ifM (liftIO $ checkSafeDropProofEndTime (Just proof))
+			( dropaction proof
+			, do
+				safeDropProofExpired
+				nodropaction
+			)
 
 data DropException = DropException SomeException
 	deriving (Typeable, Show)
diff --git a/Annex/Proxy.hs b/Annex/Proxy.hs
index 167a76ca47..4563579ef2 100644
--- a/Annex/Proxy.hs
+++ b/Annex/Proxy.hs
@@ -96,7 +96,7 @@ proxySpecialRemote protoversion r ihdl ohdl owaitv endv = go
 			liftIO $ sendmessage FAILURE
 			go
 		Just (REMOVE k) -> do
-			tryNonAsync (Remote.removeKey r k) >>= \case
+			tryNonAsync (Remote.removeKey r Nothing k) >>= \case
 				Right () -> liftIO $ sendmessage SUCCESS
 				Left err -> liftIO $ propagateerror err
 			go
diff --git a/Annex/SafeDropProof.hs b/Annex/SafeDropProof.hs
new file mode 100644
index 0000000000..b54b627cbc
--- /dev/null
+++ b/Annex/SafeDropProof.hs
@@ -0,0 +1,34 @@
+{- git-annex safe drop proof
+ -
+ - Copyright 2014-2024 Joey Hess <id@joeyh.name>
+ -
+ - Licensed under the GNU AGPL version 3 or higher.
+ -}
+
+{-# LANGUAGE OverloadedStrings #-}
+
+module Annex.SafeDropProof (
+	SafeDropProof,
+	safeDropProofEndTime,
+	safeDropProofExpired,
+	checkSafeDropProofEndTime,
+) where
+
+import Annex.Common
+import Types.NumCopies
+
+import Data.Time.Clock.POSIX
+
+safeDropProofExpired :: Annex ()
+safeDropProofExpired = do
+	showNote "unsafe"
+	showLongNote $ UnquotedString
+		"Dropping took too long, and locks on remotes may have expired."
+
+checkSafeDropProofEndTime :: Maybe SafeDropProof -> IO Bool
+checkSafeDropProofEndTime p = case safeDropProofEndTime =<< p of
+	Nothing -> return True
+	Just t -> do
+		now <- getPOSIXTime
+		return (t < now)
+
diff --git a/CmdLine/GitRemoteAnnex.hs b/CmdLine/GitRemoteAnnex.hs
index 612e78691e..ae06d76d7a 100644
--- a/CmdLine/GitRemoteAnnex.hs
+++ b/CmdLine/GitRemoteAnnex.hs
@@ -996,7 +996,7 @@ dropKey rmt k = tryNonAsync (dropKey' rmt k) >>= \case
 
 dropKey' :: Remote -> Key -> Annex ()
 dropKey' rmt k = getKeyExportLocations rmt k >>= \case
-	Nothing -> Remote.removeKey rmt k
+	Nothing -> Remote.removeKey rmt Nothing k
 	Just locs -> forM_ locs $ \loc -> 
 		Remote.removeExport (Remote.exportActions rmt) k loc
 
diff --git a/Command/Drop.hs b/Command/Drop.hs
index 80908c1923..54815ce20c 100644
--- a/Command/Drop.hs
+++ b/Command/Drop.hs
@@ -151,7 +151,7 @@ performRemote pcc key afile numcopies mincopies remote ud = do
 				, "proof:"
 				, show proof
 				]
-			ok <- Remote.action (Remote.removeKey remote key)
+			ok <- Remote.action (Remote.removeKey remote proof key)
 			next $ cleanupRemote key remote ud ok
 		, stop
 		)
diff --git a/Command/Fsck.hs b/Command/Fsck.hs
index 0acb018718..3688ef6184 100644
--- a/Command/Fsck.hs
+++ b/Command/Fsck.hs
@@ -639,7 +639,7 @@ badContentRemote remote localcopy key = do
 					)
 		)
 
-	dropped <- tryNonAsync (Remote.removeKey remote key)
+	dropped <- tryNonAsync (Remote.removeKey remote Nothing key)
 	when (isRight dropped) $
 		Remote.logStatus remote key InfoMissing
 	return $ case (movedbad, dropped) of
diff --git a/Command/Move.hs b/Command/Move.hs
index 1abdeb8ca0..ffc58e0120 100644
--- a/Command/Move.hs
+++ b/Command/Move.hs
@@ -296,23 +296,26 @@ fromPerform' present updatelocationlog src key afile = do
 fromDrop :: Remote -> UUID -> DestStartedWithCopy -> Key -> AssociatedFile -> ([UnVerifiedCopy] -> [UnVerifiedCopy])-> CommandPerform
 fromDrop src destuuid deststartedwithcopy key afile adjusttocheck =
 	willDropMakeItWorse (Remote.uuid src) destuuid deststartedwithcopy key afile >>= \case
-		DropAllowed -> dropremote "moved"
+		DropAllowed -> dropremote Nothing "moved"
 		DropCheckNumCopies -> do
 			(numcopies, mincopies) <- getSafestNumMinCopies afile key
 			(tocheck, verified) <- verifiableCopies key [Remote.uuid src]
 			verifyEnoughCopiesToDrop "" key (Just (Remote.uuid src)) Nothing numcopies mincopies [Remote.uuid src] verified
-				(adjusttocheck tocheck) (dropremote . showproof) faileddropremote
+				(adjusttocheck tocheck) dropremotewithproof faileddropremote
 		DropWorse -> faileddropremote
   where
 	showproof proof = "proof: " ++ show proof
 
-	dropremote reason = do
+	dropremotewithproof proof = 
+		dropremote (Just proof) (showproof proof)
+
+	dropremote mproof reason = do
 		fastDebug "Command.Move" $ unwords
 			[ "Dropping from remote"
 			, show src
 			, "(" ++ reason ++ ")"
 			]
-		ok <- Remote.action (Remote.removeKey src key)
+		ok <- Remote.action (Remote.removeKey src mproof key)
 		when ok $
 			logMoveCleanup deststartedwithcopy
 		next $ Command.Drop.cleanupRemote key src (Command.Drop.DroppingUnused False) ok
diff --git a/Command/TestRemote.hs b/Command/TestRemote.hs
index f0f2ac8efe..582323d70b 100644
--- a/Command/TestRemote.hs
+++ b/Command/TestRemote.hs
@@ -303,7 +303,7 @@ test runannex mkr mkk =

(Diff truncated)

Changed pages:

todo/git-annex proxies
Changed by: joeyh
Commit type: git
Date: 21:04:12 07/03/24

status

diff --git a/doc/todo/git-annex_proxies.mdwn b/doc/todo/git-annex_proxies.mdwn
index fe24199d0e..4c33d87abb 100644
--- a/doc/todo/git-annex_proxies.mdwn
+++ b/doc/todo/git-annex_proxies.mdwn
@@ -33,7 +33,7 @@ Planned schedule of work:
   right in the http protocol from the beginning.
 
   Status: Content retention files implemented. P2P LOCKCONTENT uses a 10
-  minute retention in case it gets killed. Need to implement PRE-REMOVE.
+  minute retention in case it gets killed. P2P protocol changes started.
 
 * websockets or something else for LOCKCONTENT over http?

Changed pages:

design/p2p protocol over http/draft1

todo/P2P locking connection drop safety
Changed by: joeyh
Commit type: git
Date: 20:59:22 07/03/24

REMOVE-BEFORE and GETTIMESTAMP
Only implemented server side, not used client side yet.
And not yet implemented for proxies/clusters, for which there's a build
warning about unhandled cases.
This is P2P protocol version 3. Probably will be the only change in that
version..
Added a dependency on clock to access a monotonic clock.
On i386-ancient, that is at version 0.2.0.0.

diff --git a/CHANGELOG b/CHANGELOG
index 9e467cfb12..c14e78ed55 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -2,6 +2,10 @@ git-annex (10.20240702) UNRELEASED; urgency=medium
 
   * assistant: Fix a race condition that could cause a pointer file to
     get ingested into the annex.
+  * Avoid potential data loss in situations where git-annex-shell or
+    git-annex remotedaemon is killed while locking a key to prevent its
+    removal.
+  * Added a dependency on clock.
 
  -- Joey Hess <id@joeyh.name>  Tue, 02 Jul 2024 12:14:53 -0400
 
diff --git a/P2P/Annex.hs b/P2P/Annex.hs
index f689ae8d96..40152f2839 100644
--- a/P2P/Annex.hs
+++ b/P2P/Annex.hs
@@ -110,15 +110,24 @@ runLocal runst runner a = case a of
 		case v of
 			Left e -> return $ Left $ ProtoFailureException e
 			Right result -> runner (next result)
-	RemoveContent k next -> do
+	RemoveContent k mts next -> do
 		let cleanup = do
 			logStatus k InfoMissing
 			return True
+		let checkts = case mts of
+			Nothing -> return True
+			Just ts -> do
+				now <- liftIO getMonotonicTimestampIO
+				return (now < ts)
 		v <- tryNonAsync $
 			ifM (Annex.Content.inAnnex k)
-				( lockContentForRemoval k cleanup $ \contentlock -> do
-					removeAnnex contentlock
-					cleanup
+				( lockContentForRemoval k cleanup $ \contentlock ->
+					ifM checkts
+						( do
+							removeAnnex contentlock
+							cleanup
+						, return False
+						)
 				, return True
 				)
 		case v of
diff --git a/P2P/IO.hs b/P2P/IO.hs
index 14c588dac5..aa52973b61 100644
--- a/P2P/IO.hs
+++ b/P2P/IO.hs
@@ -25,6 +25,7 @@ module P2P.IO
 	, describeProtoFailure
 	, runNetProto
 	, runNet
+	, getMonotonicTimestampIO
 	) where
 
 import Common
@@ -53,6 +54,11 @@ import qualified Data.ByteString as B
 import qualified Data.ByteString.Lazy as L
 import qualified Network.Socket as S
 import System.PosixCompat.Files (groupReadMode, groupWriteMode, otherReadMode, otherWriteMode)
+#if MIN_VERSION_clock(0,3,0)
+import qualified System.Clock as Clock
+#else
+import qualified System.Posix.Clock as Clock
+#endif
 
 -- Type of interpreters of the Proto free monad.
 type RunProto m = forall a. Proto a -> m (Either ProtoFailure a)
@@ -282,6 +288,8 @@ runNet runst conn runner f = case f of
 		runner next
 	GetProtocolVersion next ->
 		liftIO (readTVarIO versiontvar) >>= runner . next
+	GetMonotonicTimestamp next ->
+		liftIO getMonotonicTimestampIO >>= runner . next
   where
 	-- This is only used for running Net actions when relaying,
 	-- so it's ok to use runNetProto, despite it not supporting
@@ -452,3 +460,7 @@ relayReader v hout = loop
 			else getsome (b:bs)
 	
 	chunk = 65536
+
+getMonotonicTimestampIO :: IO MonotonicTimestamp
+getMonotonicTimestampIO = (MonotonicTimestamp . fromIntegral . Clock.sec)
+	<$> Clock.getTime Clock.Monotonic
diff --git a/P2P/Protocol.hs b/P2P/Protocol.hs
index 79d8fbd8a3..da66e6aaed 100644
--- a/P2P/Protocol.hs
+++ b/P2P/Protocol.hs
@@ -56,7 +56,7 @@ defaultProtocolVersion :: ProtocolVersion
 defaultProtocolVersion = ProtocolVersion 0
 
 maxProtocolVersion :: ProtocolVersion
-maxProtocolVersion = ProtocolVersion 2
+maxProtocolVersion = ProtocolVersion 3
 
 newtype ProtoAssociatedFile = ProtoAssociatedFile AssociatedFile
 	deriving (Show)
@@ -71,6 +71,9 @@ data Validity = Valid | Invalid
 newtype Bypass = Bypass (S.Set UUID)
 	deriving (Show, Monoid, Semigroup)
 
+newtype MonotonicTimestamp = MonotonicTimestamp Integer
+	deriving (Show, Eq, Ord)
+
 -- | Messages in the protocol. The peer that makes the connection
 -- always initiates requests, and the other peer makes responses to them.
 data Message
@@ -86,6 +89,8 @@ data Message
 	| LOCKCONTENT Key
 	| UNLOCKCONTENT
 	| REMOVE Key
+	| REMOVE_BEFORE MonotonicTimestamp Key
+	| GETTIMESTAMP
 	| GET Offset ProtoAssociatedFile Key
 	| PUT ProtoAssociatedFile Key
 	| PUT_FROM Offset
@@ -98,6 +103,7 @@ data Message
 	| BYPASS Bypass
 	| DATA Len -- followed by bytes of data
 	| VALIDITY Validity
+	| TIMESTAMP MonotonicTimestamp
 	| ERROR String
 	deriving (Show)
 
@@ -114,6 +120,8 @@ instance Proto.Sendable Message where
 	formatMessage (LOCKCONTENT key) = ["LOCKCONTENT", Proto.serialize key]
 	formatMessage UNLOCKCONTENT = ["UNLOCKCONTENT"]
 	formatMessage (REMOVE key) = ["REMOVE", Proto.serialize key]
+	formatMessage (REMOVE_BEFORE ts key) = ["REMOVE-BEFORE", Proto.serialize ts, Proto.serialize key]
+	formatMessage GETTIMESTAMP = ["GETTIMESTAMP"]
 	formatMessage (GET offset af key) = ["GET", Proto.serialize offset, Proto.serialize af, Proto.serialize key]
 	formatMessage (PUT af key) = ["PUT", Proto.serialize af, Proto.serialize key]
 	formatMessage (PUT_FROM offset) = ["PUT-FROM", Proto.serialize offset]
@@ -124,9 +132,10 @@ instance Proto.Sendable Message where
 	formatMessage FAILURE = ["FAILURE"]
 	formatMessage (FAILURE_PLUS uuids) = ("FAILURE-PLUS":map Proto.serialize uuids)
 	formatMessage (BYPASS (Bypass uuids)) = ("BYPASS":map Proto.serialize (S.toList uuids))
+	formatMessage (DATA len) = ["DATA", Proto.serialize len]
 	formatMessage (VALIDITY Valid) = ["VALID"]
 	formatMessage (VALIDITY Invalid) = ["INVALID"]
-	formatMessage (DATA len) = ["DATA", Proto.serialize len]
+	formatMessage (TIMESTAMP ts) = ["TIMESTAMP", Proto.serialize ts]
 	formatMessage (ERROR err) = ["ERROR", Proto.serialize err]
 
 instance Proto.Receivable Message where
@@ -142,6 +151,8 @@ instance Proto.Receivable Message where
 	parseCommand "LOCKCONTENT" = Proto.parse1 LOCKCONTENT
 	parseCommand "UNLOCKCONTENT" = Proto.parse0 UNLOCKCONTENT
 	parseCommand "REMOVE" = Proto.parse1 REMOVE
+	parseCommand "REMOVE-BEFORE" = Proto.parse2 REMOVE_BEFORE
+	parseCommand "GETTIMESTAMP" = Proto.parse0 GETTIMESTAMP
 	parseCommand "GET" = Proto.parse3 GET
 	parseCommand "PUT" = Proto.parse2 PUT
 	parseCommand "PUT-FROM" = Proto.parse1 PUT_FROM
@@ -153,9 +164,10 @@ instance Proto.Receivable Message where
 	parseCommand "FAILURE-PLUS" = Proto.parseList FAILURE_PLUS
 	parseCommand "BYPASS" = Proto.parseList (BYPASS . Bypass . S.fromList)
 	parseCommand "DATA" = Proto.parse1 DATA
-	parseCommand "ERROR" = Proto.parse1 ERROR
 	parseCommand "VALID" = Proto.parse0 (VALIDITY Valid)
 	parseCommand "INVALID" = Proto.parse0 (VALIDITY Invalid)
+	parseCommand "TIMESTAMP" = Proto.parse1 TIMESTAMP
+	parseCommand "ERROR" = Proto.parse1 ERROR
 	parseCommand _ = Proto.parseFail
 
 instance Proto.Serializable ProtocolVersion where
@@ -170,6 +182,10 @@ instance Proto.Serializable Len where
 	serialize (Len n) = show n
 	deserialize = Len <$$> readish
 
+instance Proto.Serializable MonotonicTimestamp where
+	serialize (MonotonicTimestamp n) = show n
+	deserialize = MonotonicTimestamp <$$> readish
+
 instance Proto.Serializable Service where
 	serialize UploadPack = "git-upload-pack"
 	serialize ReceivePack = "git-receive-pack"
@@ -249,6 +265,7 @@ data NetF c
 	| SetProtocolVersion ProtocolVersion c
 	--- ^ Called when a new protocol version has been negotiated.
 	| GetProtocolVersion (ProtocolVersion -> c)
+	| GetMonotonicTimestamp (MonotonicTimestamp -> c)
 	deriving (Functor)
 
 type Net = Free NetF
@@ -294,9 +311,11 @@ data LocalF c
 	| SetPresent Key UUID c
 	| CheckContentPresent Key (Bool -> c)
 	-- ^ Checks if the whole content of the key is locally present.
-	| RemoveContent Key (Bool -> c)
+	| RemoveContent Key (Maybe MonotonicTimestamp) (Bool -> c)
 	-- ^ If the content is not present, still succeeds.
 	-- May fail if not enough copies to safely drop, etc.
+	-- After locking the content for removal, checks if it's later
+	-- than the MonotonicTimestamp, and fails.

(Diff truncated)

Changed pages:

bugs/assistant (webapp) commited unlocked link to annex/comment 8 690015a440c371ff330530d01ba80de6
Changed by: yarikoptic
Commit type: web
Date: 20:42:11 07/03/24

Added a comment

diff --git a/doc/bugs/assistant___40__webapp__41___commited_unlocked_link_to_annex/comment_8_690015a440c371ff330530d01ba80de6._comment b/doc/bugs/assistant___40__webapp__41___commited_unlocked_link_to_annex/comment_8_690015a440c371ff330530d01ba80de6._comment
new file mode 100644
index 0000000000..7f3f13b661
--- /dev/null
+++ b/doc/bugs/assistant___40__webapp__41___commited_unlocked_link_to_annex/comment_8_690015a440c371ff330530d01ba80de6._comment
@@ -0,0 +1,8 @@
+[[!comment format=mdwn
+ username="yarikoptic"
+ avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
+ subject="comment 8"
+ date="2024-07-03T20:42:11Z"
+ content="""
+well, I would need a script to recover since it is not a matter of a single file -- IIRC there was a good number of them.  But then we get into \"why not to make it auto-recover\" then.
+"""]]

Changed pages:

todo/P2P locking connection drop safety
Changed by: joeyh
Commit type: git
Date: 19:54:14 07/03/24

Merge branch 'master' into p2p_locking

Changed pages:

todo/P2P locking connection drop safety
Changed by: joeyh
Commit type: git
Date: 19:53:25 07/03/24

update

diff --git a/doc/todo/P2P_locking_connection_drop_safety.mdwn b/doc/todo/P2P_locking_connection_drop_safety.mdwn
index 209ee18379..a45002030e 100644
--- a/doc/todo/P2P_locking_connection_drop_safety.mdwn
+++ b/doc/todo/P2P_locking_connection_drop_safety.mdwn
@@ -17,6 +17,7 @@ I'm inclined to agree with past me. While the P2P protocol could be
 extended with a way to verify that the connection is still open, there
 is a point where git-annex has told the remote to drop, and is relying on
 the locks remaining locked until the drop finishes.
+--[[Joey]]
 
 Worst case, I can imagine that the local git-annex process takes the remote
 locks. Then it's put to sleep for a day. Then it wakes up and drops from
@@ -24,11 +25,11 @@ the other remote. The P2P connections for the locks have long since closed.
 Consider for example, a ssh password prompt on connection to the remote to
 drop the content, and the user taking a long time to respond.
 
-It seems that lockContentWhile needs to guarantee that the content remains
+It seems that LOCKCONTENT needs to guarantee that the content remains
 locked for some amount of time. Then local git-annex would know it
 has at most that long to drop the content. But it's the remote that's
 dropping that really needs to know. So, extend the P2P protocol with a
-PRE-REMOVE step. After receiving PRE-REMOVE N, a REMOVE of that key is only
+PRE-REMOVE step. After receiving PRE-REMOVE N Key, a REMOVE of that key is only
 allowed until N seconds later. Sending PRE-REMOVE first, followed by
 LOCKCONTENT will guarantee the content remains locked for the full amount
 of time.
@@ -62,4 +63,51 @@ git-annex gets installed, a user is likely to have been using git-annex
 
 OTOH putting the timestamp in the lock file may be hard (eg on Windows).
 
---[[Joey]]
+> Status: Content retention files implemented on `p2p_locking` branch.
+> P2P LOCKCONTENT uses a 10 minute retention in case it gets killed,
+> but other values can be used in the future safely.
+
+----
+
+Extending the P2P protocol is a bit tricky, because the same P2P
+protocol connection could be used for several different things at
+the same time. A PRE-REMOVE N Key might be followed by removals of other
+keys, and eventually a removal of the requested key. There are
+sometimes pools of P2P connections that get used like this.
+So the server would need to cache some number of PRE-REMOVE timestamps.
+How many?
+
+Certainly care would need to be taken to send PRE-REMOVE to the same
+connection as REMOVE. How?
+
+Could this be done without extending the REMOVE side of the P2P protocol?
+
+1. check start time
+2. LOCKCONTENT
+3. prepare to remove
+4. in checkVerifiedCopy, 
+   check current time.. fail if more than 10 minutes from start
+5. REMOVE
+
+The issue with this is that git-annex could be paused for any amount of
+time between steps 4 and 5. Usually it won't pause.. 
+mkSafeDropProof calls checkVerifiedCopy and constructs the proof,
+and then it immediately sends REMOVE. But of course sending REMOVE
+could take arbitrarily long. Or git-annex could be paused at just the wrong
+point.
+
+Ok, let's reconsider... Add GETTIMESTAMP which causes the server to
+return its current timestamp. The same timestamp must be returned on any
+connection to the server, eg the server must have a single clock.
+That can be called before LOCKCONTENT.
+Then REMOVE Key Timestamp can fail if the current time is past the
+specified timestamp. 
+
+How to handle this when proxying to a cluster? In a cluster, each node
+has a different clock. So GETTIMESTAMP will return a bunch of times.
+The cluster can get its own current time, and return that to the client.
+Then REMOVE Key Timestamp can have the timestamp adjusted when it's sent
+out to each client, by calling GETTIMESTAMP again and applying the offsets
+between the cluster's clock and each node's clock.
+
+This approach would need to use a monotonic clock!

Changed pages:

todo/git-annex proxies
Changed by: joeyh
Commit type: git
Date: 19:01:34 07/03/24

status

diff --git a/doc/todo/git-annex_proxies.mdwn b/doc/todo/git-annex_proxies.mdwn
index 0085fd5b98..fe24199d0e 100644
--- a/doc/todo/git-annex_proxies.mdwn
+++ b/doc/todo/git-annex_proxies.mdwn
@@ -32,6 +32,9 @@ Planned schedule of work:
   because it will involve protocol changes and we need to get locking
   right in the http protocol from the beginning.
 
+  Status: Content retention files implemented. P2P LOCKCONTENT uses a 10
+  minute retention in case it gets killed. Need to implement PRE-REMOVE.
+
 * websockets or something else for LOCKCONTENT over http?
 
 * will client notice promptly when http connection with server

Changed pages:

todo/P2P locking connection drop safety
Changed by: joeyh
Commit type: git
Date: 18:44:38 07/03/24

add content retention files
This allows lockContentShared to lock content for eg, 10 minutes and
if the process then gets terminated before it can unlock, the content
will remain locked for that amount of time.
The Windows implementation is not yet tested.
In P2P.Annex, a duration of 10 minutes is used. This way, when p2pstdio
or remotedaemon is serving the P2P protocol, and is asked to
LOCKCONTENT, and that process gets killed, the content will not be
subject to deletion. This is not a perfect solution to
doc/todo/P2P_locking_connection_drop_safety.mdwn yet, but it gets most
of the way there, without needing any P2P protocol changes.
This is only done in v10 and higher repositories (or on Windows). It
might be possible to backport it to v8 or earlier, but it would
complicate locking even further, and without a separate lock file, might
be hard. I think that by the time this fix reaches a given user, they
will probably have been running git-annex 10.x long enough that their v8
repositories will have upgraded to v10 after the 1 year wait. And it's
not as if git-annex hasn't already been subject to this problem (though
I have not heard of any data loss caused by it) for 6 years already, so
waiting another fraction of a year on top of however long it takes this
fix to reach users is unlikely to be a problem.

diff --git a/Annex/Content.hs b/Annex/Content.hs
index 3c10def782..2a52b59400 100644
--- a/Annex/Content.hs
+++ b/Annex/Content.hs
@@ -1,6 +1,6 @@
 {- git-annex file content managing
  -
- - Copyright 2010-2023 Joey Hess <id@joeyh.name>
+ - Copyright 2010-2024 Joey Hess <id@joeyh.name>
  -
  - Licensed under the GNU AGPL version 3 or higher.
  -}
@@ -88,6 +88,7 @@ import Git.FilePath
 import Annex.Perms
 import Annex.Link
 import Annex.LockPool
+import Annex.LockFile
 import Annex.UUID
 import Annex.InodeSentinal
 import Annex.ReplaceFile
@@ -103,6 +104,8 @@ import Logs.Location
 import Utility.InodeCache
 import Utility.CopyFile
 import Utility.Metered
+import Utility.HumanTime
+import Utility.TimeStamp
 #ifndef mingw32_HOST_OS
 import Utility.FileMode
 #endif
@@ -110,38 +113,102 @@ import qualified Utility.RawFilePath as R
 
 import qualified System.FilePath.ByteString as P
 import System.PosixCompat.Files (isSymbolicLink, linkCount)
+import Data.Time.Clock.POSIX
 
 {- Prevents the content from being removed while the action is running.
  - Uses a shared lock.
  -
  - If locking fails, or the content is not present, throws an exception
  - rather than running the action.
+ -
+ - When a Duration is provided, the content is prevented from being removed
+ - for that amount of time, even if the current process is terminated.
+ - (This is only done when using a separate lock file from the content
+ - file eg in v10 and higher repositories.)
  -}
-lockContentShared :: Key -> (VerifiedCopy -> Annex a) -> Annex a
-lockContentShared key a = lockContentUsing lock key notpresent $
-	ifM (inAnnex key)
-		( do
-			u <- getUUID
-			withVerifiedCopy LockedCopy u (return True) a
-		, notpresent
-		)
+lockContentShared :: Key -> Maybe Duration -> (VerifiedCopy -> Annex a) -> Annex a
+lockContentShared key mduration a = do
+	retention <- case mduration of
+		Nothing -> pure Nothing
+		Just duration -> do
+			rt <- calcRepo (gitAnnexContentRetentionTimestamp key)
+			now <- liftIO getPOSIXTime
+			pure $ Just
+				( rt
+				, now + fromIntegral (durationSeconds duration)
+				)
+	lockContentUsing (lock retention) key notpresent $
+		ifM (inAnnex key)
+			( do
+				u <- getUUID
+				withVerifiedCopy LockedCopy u (return True) a
+			, notpresent
+			)
   where
 	notpresent = giveup $ "failed to lock content: not present"
 #ifndef mingw32_HOST_OS
-	lock _ (Just lockfile) = 
-		( posixLocker tryLockShared lockfile
-		, Just (posixLocker tryLockExclusive lockfile)
+	lock retention _ (Just lockfile) = 
+		( posixLocker tryLockShared lockfile >>= \case
+			Just lck -> do
+				writeretention retention
+				return (Just lck)
+			Nothing -> return Nothing
+		, Just $ posixLocker tryLockExclusive lockfile >>= \case
+			Just lck -> do
+				dropretention retention
+				return (Just lck)
+			Nothing -> return Nothing
 		)
-	lock contentfile Nothing =
+	lock _ contentfile Nothing =
 		( tryLockShared Nothing contentfile
 		, Nothing
 		)
 #else
-	lock = winLocker lockShared
+	lock retention v = 
+		let (locker, postunlock) = winLocker lockShared v
+		in 
+			( locker >>= \case
+				Just lck -> do
+					writeretention retention
+					return (Just lck)
+				Nothing -> return Nothing
+			, \lckfile -> do
+				maybe noop (\a -> a lckfile) postunlock
+				lockdropretention retention
+			)
+
+	lockdropretention Nothing = noop
+	lockdropretention retention@(Just _) =
+		-- In order to dropretention, have to
+		-- take an exclusive lock.
+		let (exlocker, expostunlock) =
+			winLocker lockExclusive v
+		exlocker >>= \case
+			Nothing -> noop
+			Just lck -> do
+				dropretention retention
+				liftIO $ dropLock lck
+		fromMaybe noop expostunlock
 #endif
+	
+	writeretention Nothing = noop
+	writeretention (Just (rt, retentionts)) = 
+		writeContentRetentionTimestamp key rt retentionts
+	
+	-- When this is called, an exclusive lock has been taken, so no other
+	-- processes can be writing to the retention time stamp file.
+	-- The timestamp in the file may have been written by this
+	-- call to lockContentShared or a later call. Only delete the file
+	-- in the former case.
+	dropretention Nothing = noop
+	dropretention (Just (rt, retentionts)) =
+		readContentRetentionTimestamp rt >>= \case
+			Just ts | ts == retentionts ->
+				removeRetentionTimeStamp key rt
+			_ -> noop
 
-{- Exclusively locks content, while performing an action that
- - might remove it.
+{- Exclusively locks content, including checking the retention timestamp, 
+ - while performing an action that might remove it.
  -
  - If locking fails, throws an exception rather than running the action.
  -
@@ -155,7 +222,11 @@ lockContentForRemoval key fallback a = lockContentUsing lock key fallback $
 	a (ContentRemovalLock key)
   where
 #ifndef mingw32_HOST_OS
-	lock _ (Just lockfile) = (posixLocker tryLockExclusive lockfile, Nothing)
+	lock _ (Just lockfile) =
+		( checkRetentionTimestamp key
+			(posixLocker tryLockExclusive lockfile)
+		, Nothing
+		)
 	{- No lock file, so the content file itself is locked. 
 	 - Since content files are stored with the write bit
 	 - disabled, have to fiddle with permissions to open
@@ -167,12 +238,30 @@ lockContentForRemoval key fallback a = lockContentUsing lock key fallback $
 			(tryLockExclusive Nothing contentfile)
 		in (lck, Nothing)
 #else
-	lock = winLocker lockExclusive
+	lock = checkRetentionTimestamp key
+		(winLocker lockExclusive)
 #endif
 
 {- Passed the object content file, and maybe a separate lock file to use,
  - when the content file itself should not be locked. -}
-type ContentLocker = RawFilePath -> Maybe LockFile -> (Annex (Maybe LockHandle), Maybe (Annex (Maybe LockHandle)))
+type ContentLocker 
+	= RawFilePath
+	-> Maybe LockFile 
+	->
+		( Annex (Maybe LockHandle)
+		-- ^ Takes the lock, which may be shared or exclusive.
+#ifndef mingw32_HOST_OS
+		, Maybe (Annex (Maybe LockHandle))
+		-- ^ When the above takes a shared lock, this is used
+		-- to take an exclusive lock, after dropping the shared lock,
+		-- and prior to deleting the lock file, in order to 
+		-- ensure that no other processes also have a shared lock.
+#else
+		, Maybe (RawFilePath -> Annex ())
+		-- ^ On Windows, this is called after the lock is dropped,
+		-- but before the lock file is cleaned up.
+#endif
+		)
 
 #ifndef mingw32_HOST_OS
 posixLocker :: (Maybe ModeSetter -> LockFile -> Annex (Maybe LockHandle)) -> LockFile -> Annex (Maybe LockHandle)
@@ -264,13 +353,17 @@ lockContentUsing contentlocker key fallback a = withContentLockFile key $ \mlock
 			maybe noop cleanuplockfile mlockfile
 			liftIO $ dropLock lck
 #else
-	unlock _ mlockfile lck = do
+	unlock postunlock mlockfile lck = do
 		-- Can't delete a locked file on Windows,

(Diff truncated)

Changed pages:

todo/P2P locking connection drop safety

todo/git-annex proxies
Changed by: joeyh
Commit type: git
Date: 17:15:09 07/03/24

todo

diff --git a/doc/todo/P2P_locking_connection_drop_safety.mdwn b/doc/todo/P2P_locking_connection_drop_safety.mdwn
new file mode 100644
index 0000000000..209ee18379
--- /dev/null
+++ b/doc/todo/P2P_locking_connection_drop_safety.mdwn
@@ -0,0 +1,65 @@
+The P2P protocol's LOCKCONTENT assumes that the P2P connection does not get
+closed unexpectedly. If the P2P connection does close before the drop
+happens, the remote's lock will be released, but the git-annex that is
+doing the dropping does not have a way to find that out.
+
+This in particular affects drops from remotes. Drops from the local
+repository have a ContentRemovalLock that doesn't have this problem.
+
+This was discussed in [[!commit 73a6b9b51455f2ae8483a86a98e9863fffe9ebac]]
+(2016). There I concluded:
+
+	Probably this needs to be fixed by eg, making lockContentWhile catch any
+	exceptions due to the connection closing, and in that case, wait a
+	significantly long time before dropping the lock.
+
+I'm inclined to agree with past me. While the P2P protocol could be
+extended with a way to verify that the connection is still open, there
+is a point where git-annex has told the remote to drop, and is relying on
+the locks remaining locked until the drop finishes.
+
+Worst case, I can imagine that the local git-annex process takes the remote
+locks. Then it's put to sleep for a day. Then it wakes up and drops from
+the other remote. The P2P connections for the locks have long since closed.
+Consider for example, a ssh password prompt on connection to the remote to
+drop the content, and the user taking a long time to respond.
+
+It seems that lockContentWhile needs to guarantee that the content remains
+locked for some amount of time. Then local git-annex would know it
+has at most that long to drop the content. But it's the remote that's
+dropping that really needs to know. So, extend the P2P protocol with a
+PRE-REMOVE step. After receiving PRE-REMOVE N, a REMOVE of that key is only
+allowed until N seconds later. Sending PRE-REMOVE first, followed by
+LOCKCONTENT will guarantee the content remains locked for the full amount
+of time.
+
+How long? 10 minutes is arbitrary, but seems in the right ballpark. Since
+this will cause drops to fail if they timeout sitting at a ssh password
+prompt, it needs to be more than a few minutes. But making it too long, eg
+an hour can result in content being stuck locked on a remote for a long
+time, preventing a later legitimate drop. It could be made configurable, if
+needed, by extending the P2P protocol so LOCKCONTENT was passed the amount
+of time.
+
+Having lockContentWhile catch all exceptions and keep the content locked
+for the time period won't work though. Systemd reaps processes on ssh
+connection close. And if the P2P protocol is served by `git annex
+remotedaemon` for tor, or something similar for future P2P over HTTP
+(either a HTTP daemon or a CGI script), nothing guarantees that such a
+process is kept running. An admin may bounce the HTTP server at any point,
+or the whole system reboot.
+
+----
+
+So, this needs a way to make lockContentShared guarentee it remains
+locked for an amount of time even after the process has exited.
+
+In a v10 repo, the content lock file is separate from the content file,
+and it is currently an empty file. So a timestamp could be put in there.
+It seems ok to only fix this in v10, because by the time the fixed
+git-annex gets installed, a user is likely to have been using git-annex
+10.x long enough (1 year) for their repo to have been upgraded to v10.
+
+OTOH putting the timestamp in the lock file may be hard (eg on Windows).
+
+--[[Joey]]
diff --git a/doc/todo/git-annex_proxies.mdwn b/doc/todo/git-annex_proxies.mdwn
index ddb1ea92b9..0085fd5b98 100644
--- a/doc/todo/git-annex_proxies.mdwn
+++ b/doc/todo/git-annex_proxies.mdwn
@@ -28,6 +28,10 @@ Planned schedule of work:
 
 ## work notes
 
+* [[todo/P2P_locking_connection_drop_safety]] is blocking http protocol,
+  because it will involve protocol changes and we need to get locking
+  right in the http protocol from the beginning.
+
 * websockets or something else for LOCKCONTENT over http?
 
 * will client notice promptly when http connection with server

Changed pages:

todo/git-annex proxies
Changed by: joeyh
Commit type: git
Date: 22:04:29 07/02/24

update

diff --git a/doc/todo/git-annex_proxies.mdwn b/doc/todo/git-annex_proxies.mdwn
index d037124a69..ddb1ea92b9 100644
--- a/doc/todo/git-annex_proxies.mdwn
+++ b/doc/todo/git-annex_proxies.mdwn
@@ -33,6 +33,13 @@ Planned schedule of work:
 * will client notice promptly when http connection with server
   is closed in LOCKCONTENT?
 
+* Should integrate into an existing web server similarly to git
+  smart http server. May as well also support it as a standalone
+  webserver.
+
+* endpoint versioning should include v1 and v0 to support connections
+  proxied from older clients
+
 ## items deferred until later for [[design/passthrough_proxy]]
 
 * Check annex.diskreserve when proxying for special remotes

Changed pages:

todo/git-annex proxies
Changed by: joeyh
Commit type: git
Date: 20:16:37 07/02/24

update

diff --git a/doc/todo/git-annex_proxies.mdwn b/doc/todo/git-annex_proxies.mdwn
index 63366bfee6..d037124a69 100644
--- a/doc/todo/git-annex_proxies.mdwn
+++ b/doc/todo/git-annex_proxies.mdwn
@@ -28,14 +28,17 @@ Planned schedule of work:
 
 ## work notes
 
-For June's work on [[design/passthrough_proxy]], remaining todos:
+* websockets or something else for LOCKCONTENT over http?
+
+* will client notice promptly when http connection with server
+  is closed in LOCKCONTENT?
+
+## items deferred until later for [[design/passthrough_proxy]]
 
 * Check annex.diskreserve when proxying for special remotes
   to avoid the proxy's disk filling up with the temporary object file
   cached there.
 
-## items deferred until later for [[design/passthrough_proxy]]
-
 * Resuming an interrupted download from proxied special remote makes the proxy
   re-download the whole content. It could instead keep some of the 
   object files around when the client does not send SUCCESS. This would

Changed pages:

design/p2p protocol over http

design/p2p protocol over http/draft1
Changed by: joeyh
Commit type: git
Date: 20:14:45 07/02/24

drafting P2P protocol over http

diff --git a/doc/design/p2p_protocol.mdwn b/doc/design/p2p_protocol.mdwn
index 392f0fbe87..b889397088 100644
--- a/doc/design/p2p_protocol.mdwn
+++ b/doc/design/p2p_protocol.mdwn
@@ -201,6 +201,9 @@ was being sent.
 
 The client replies with SUCCESS or FAILURE.
 
+Note that the client responding with SUCCESS does not indicate to the
+server that it has stored the content. It may receive it and throw it away.
+
 ## Connection to services
 
 This is used to connect to services like git-upload-pack and
diff --git a/doc/design/p2p_protocol_over_http.mdwn b/doc/design/p2p_protocol_over_http.mdwn
index 4c67f0d0bf..c402ca5d8a 100644
--- a/doc/design/p2p_protocol_over_http.mdwn
+++ b/doc/design/p2p_protocol_over_http.mdwn
@@ -72,20 +72,31 @@ needs to know that a series of requests are part of the same P2P protocol
 session. In the example above, it would not have a good way to do that.
 One solution would be to add a session identifier UUID to each request.
 
-## approach 2: HTTP API
+## approach 2: websockets
+
+The client connects to the server over a websocket. From there on,
+the protocol is encapsulated in websockets.
+
+This seems nice and simple, but again not very web native. 
+
+Some requests like `LOCKCONTENT` seem likely to need full duplex
+communication like websockets provide. But, it might be more web native to
+only use websockets for that request, and not for everything.
+
+## approach 3: HTTP API
 
 Another approach is to define a web-native API with endpoints that
 correspond to each action in the P2P protocol. 
 
 Something like this:
 
-    > GET /git-annex/v1/AUTH?clientuuid=79a5a1f4-07e8-11ef-873d-97f93ca91925 HTTP/1.0
+    > POST /git-annex/v1/AUTH?clientuuid=79a5a1f4-07e8-11ef-873d-97f93ca91925 HTTP/1.0
     < AUTH-SUCCESS ecf6d4ca-07e8-11ef-8990-9b8c1f696bf6
 
-    > GET /git-annex/v1/CHECKPRESENT?key=SHA1--foo&clientuuid=79a5a1f4-07e8-11ef-873d-97f93ca91925&serveruuid=ecf6d4ca-07e8-11ef-8990-9b8c1f696bf6 HTTP/1.0
+    > POST /git-annex/v1/CHECKPRESENT?key=SHA1--foo&clientuuid=79a5a1f4-07e8-11ef-873d-97f93ca91925&serveruuid=ecf6d4ca-07e8-11ef-8990-9b8c1f696bf6 HTTP/1.0
     > SUCCESS
 
-    > GET /git-annex/v1/PUT-FROM?key=SHA1--foo&clientuuid=79a5a1f4-07e8-11ef-873d-97f93ca91925&serveruuid=ecf6d4ca-07e8-11ef-8990-9b8c1f696bf6 HTTP/1.0
+    > POST /git-annex/v1/PUT-FROM?key=SHA1--foo&clientuuid=79a5a1f4-07e8-11ef-873d-97f93ca91925&serveruuid=ecf6d4ca-07e8-11ef-8990-9b8c1f696bf6 HTTP/1.0
     < PUT-FROM 0
 
     > POST /git-annex/v1/PUT?key=SHA1--foo&associatedfile=bar&put-from=0&clientuuid=79a5a1f4-07e8-11ef-873d-97f93ca91925&serveruuid=ecf6d4ca-07e8-11ef-8990-9b8c1f696bf6 HTTP/1.0
@@ -102,6 +113,8 @@ This needs a more complex spec. But it's easier for others to implement,
 especially since it does not need a session identifier, so the HTTP server can 
 be stateless.
 
+A full draft protocol for this is being developed at [[p2p_protocol_over_http/draft1]].
+
 ## HTTP GET
 
 It should be possible to support a regular HTTP get of a key, with
@@ -122,4 +135,11 @@ The CONNECT message allows both sides of the P2P protocol to send DATA
 messages in any order. This seems difficult to encapsulate in HTTP.
 
 Probably this can be not implemented, it's probably not needed for a HTTP
-remote?
+remote? This is used to tunnel git protocol over the P2P protocol, but for
+a HTTP remote the git repository can be accessed over HTTP as well.
+
+## security
+
+Should support HTTPS and/or be limited to only HTTPS.
+
+Authentication via http basic auth?
diff --git a/doc/design/p2p_protocol_over_http/draft1.mdwn b/doc/design/p2p_protocol_over_http/draft1.mdwn
new file mode 100644
index 0000000000..09bab25b39
--- /dev/null
+++ b/doc/design/p2p_protocol_over_http/draft1.mdwn
@@ -0,0 +1,273 @@
+[[!toc ]]
+
+Draft 1 of a complete [[P2P_protocol]] over HTTP.
+
+## git-annex protocol endpoint and version
+
+The git-annex protocol endpoint is "/git-annex" appended to the HTTP
+url of a git remote.
+
+## authentication
+
+A git-annex protocol endpoint can optionally operate in readonly mode without
+authentication.
+
+Authentication is required to make any changes.
+
+Authentication is done using HTTP basic auth. 
+
+The user is recommended to only authenticate over HTTPS, since otherwise
+HTTP basic auth (as well as git-annex data) can be snooped. But some users
+may want git-annex to use HTTP in eg a LAN.
+
+## protocol version
+
+Each request in the protocol is versioned. The versions correspond
+to P2P protocol versions, but for simplicity, the minimum version supported
+over HTTP is version 2. Every implementation of the HTTP protocol must
+support version 2.
+
+The protocol version comes before the request. Eg: `/git-annex/v2/put`
+
+If the server does not support a particular protocol version, the
+request will fail with a 404, and the client should fall back to an earlier
+protocol version, eg version 2.
+
+## common request parameters
+
+Every request has some common parameters that are always included:
+
+* `clientuuid`  
+
+  The value is the UUID of the git-annex repository of the client.
+
+* `serveruuid`
+
+  The value is the UUID of the git-annex repository that the server
+  should serve.
+
+Any request may also optionally include these parameters:
+
+* `bypass`
+
+  The value is the UUID of a cluster gateway, which the server should avoid
+  connecting to when serving a cluster. This is the equivilant of the
+  `BYPASS` message in the [[P2P_Protocol]].
+
+  This parameter can be given multiple times to list several cluster
+  gateway UUIDs.
+
+[Internally, git-annex can use these common parameters, plus the protocol
+version, to create a P2P session. The P2P session is driven through
+the AUTH, VERSION, and BYPASS messages, leaving the session ready to
+service requests.]
+
+## request messages
+
+All the requests below are sent with the HTTP POST method.
+
+### checkpresent
+
+Checks if a key is currently present on the server.
+
+Example:
+
+    > POST /git-annex/v2/checkpresent?key=SHA1--foo&clientuuid=79a5a1f4-07e8-11ef-873d-97f93ca91925&serveruuid=ecf6d4ca-07e8-11ef-8990-9b8c1f696bf6 HTTP/1.1
+    < SUCCESS
+
+There is one required additional parameter, `key`.
+
+The body of the request is empty.
+
+The server responds with "SUCCESS" if the key is present
+or "FAILURE" if it is not present.
+
+### lockcontent
+
+Locks the content of a key on the server, preventing it from being removed.
+
+There is one required additional parameter, `key`.
+
+This request opens a websocket between the client and the server.
+The server sends "SUCCESS" over the websocket once it has locked
+the content. Or it sends "FAILURE" if it is unable to lock the content.
+
+Once the server has sent "SUCCESS", the content remains locked as long as
+the client remains connected to the websocket. When the client disconnects,
+or closes the websocket, the server unlocks the content.
+
+XXX What happens if the connection times out? Will the client notice that
+in time? How does this work with P2P over ssh?
+
+### remove
+
+Remove a key's content from the server.
+
+Example:
+
+    > POST /git-annex/v2/remove?key=SHA1--foo&clientuuid=79a5a1f4-07e8-11ef-873d-97f93ca91925&serveruuid=ecf6d4ca-07e8-11ef-8990-9b8c1f696bf6 HTTP/1.1
+    < SUCCESS
+
+There is one required additional parameter, `key`.
+
+The body of the request is empty.
+
+The server responds with "SUCCESS" if the key was removed,
+or "FAILURE" if the key was not able to be removed.
+
+The server can also respond with "SUCCESS-PLUS" or "FAILURE-PLUS".

(Diff truncated)

Changed pages:

news/version 10.20231227

news/version 10.20240701
Changed by: joeyh
Commit type: git
Date: 16:31:23 07/02/24

add news item for git-annex 10.20240701

diff --git a/doc/news/version_10.20231227.mdwn b/doc/news/version_10.20231227.mdwn
deleted file mode 100644
index 580da5cbfb..0000000000
--- a/doc/news/version_10.20231227.mdwn
+++ /dev/null
@@ -1,28 +0,0 @@
-git-annex 10.20231227 released with [[!toggle text="these changes"]]
-[[!toggleable text="""  * migrate: Support distributed migrations by recording each migration,
-    and adding a --update option that updates the local repository
-    incrementally, hard linking annex objects to their new keys.
-  * pull, sync: When operating on content, automatically handle
-    distributed migrations.
-  * Added annex.syncmigrations config that can be set to false to prevent
-    pull and sync from migrating object content.
-  * migrate: Added --apply option that (re)applies all recorded
-    distributed migrations to the objects in repository.
-  * migrate: Support adding size to URL keys that were added with
-    --relaxed, by running eg: git-annex migrate --backend=URL foo
-  * When importing from a special remote, support preferred content
-    expressions that use terms that match on keys (eg "present", "copies=1").
-    Such terms are ignored when importing, since the key is not known yet.
-    Before, such expressions caused the import to fail.
-  * Support git-annex copy/move --from-anywhere --to remote.
-  * Make git-annex get/copy/move --from foo override configuration of
-    remote.foo.annex-ignore, as documented.
-  * Lower precision of timestamps in git-annex branch, which can reduce the
-    size of the branch by up to 8%.
-  * sync: Fix locking problems during merge when annex.pidlock is set.
-  * Avoid a problem with temp file names ending in "." on certian
-    filesystems that have problems with such filenames.
-  * sync, push: Avoid trying to send individual files to special remotes
-    configured with importtree=yes exporttree=no, which would always fail.
-  * Fix a crash opening sqlite databases when run in a non-unicode locale.
-    (Needs persistent-sqlite 2.13.3.)"""]]
\ No newline at end of file
diff --git a/doc/news/version_10.20240701.mdwn b/doc/news/version_10.20240701.mdwn
new file mode 100644
index 0000000000..83964ff8da
--- /dev/null
+++ b/doc/news/version_10.20240701.mdwn
@@ -0,0 +1,20 @@
+git-annex 10.20240701 released with [[!toggle text="these changes"]]
+[[!toggleable text="""  * git-annex remotes can now act as proxies that provide access to
+    their remotes. Configure this with remote.name.annex-proxy
+    and the git-annex update proxy command.
+  * Clusters are now supported. These are collections of nodes that can
+    be accessed as a single entity, accessed by one or more gateway
+    repositories.
+  * Added git-annex initcluster, updatecluster, and extendcluster commands.
+  * Fix a bug where interrupting git-annex while it is updating the
+    git-annex branch for an export could later lead to git fsck
+    complaining about missing tree objects.
+  * Tab completion of options like --from now includes special remotes,
+    as well as proxied remotes and clusters.
+  * Tab completion of many commands like info and trust now includes
+    remotes.
+  * P2P protocol version 2.
+  * Fix Windows build with Win32 2.13.4+
+    Thanks, Oleg Tolmatcev
+  * When --debugfilter or annex.debugfilter is set, avoid propigating
+    debug output from git-annex-shell, since it cannot be filtered."""]]
\ No newline at end of file

Changed pages:

bugs/assistant (webapp) commited unlocked link to annex

bugs/assistant (webapp) commited unlocked link to annex/comment 7 44ab02fd027efc56ce8bb701e5513b1d
Changed by: joeyh
Commit type: git
Date: 16:24:57 07/02/24

assistant: Fix a race condition that could cause a pointer file to get ingested into the annex
This was caused by commit fb8ab2469d389e5b1e554831eeb8b7c7a072d5d7 putting
an isPointerFile check in the wrong place. So if the file was not a pointer
file at that point, but got replaced by one before the file got locked
down, the pointer file would be ingested into the annex.
The fix is simply to move the isPointerFile check to after safeToAdd locks
down the file. Now if the file changes to a pointer file after the
isPointerFile check, ingestion will see that it changed after lockdown,
and will refuse to add it to the annex.
Sponsored-by: the NIH-funded NICEMAN (ReproNim TR&D3) project

diff --git a/Assistant/Threads/Committer.hs b/Assistant/Threads/Committer.hs
index 07013c0486..229ad17d1a 100644
--- a/Assistant/Threads/Committer.hs
+++ b/Assistant/Threads/Committer.hs
@@ -290,19 +290,34 @@ handleAdds lockdowndir havelsof largefilematcher annexdotfiles delayadd cs = ret
 		refillChanges postponed
 
 	returnWhen (null toadd) $ do
+		(addedpointerfiles, toaddrest) <- partitionEithers
+			<$> mapM checkpointerfile toadd
 		(toaddannexed, toaddsmall) <- partitionEithers
-			<$> mapM checksmall toadd
+			<$> mapM checksmall toaddrest
 		addsmall toaddsmall
 		addedannexed <- addaction toadd $
 			catMaybes <$> addannexed toaddannexed
-		return $ addedannexed ++ toaddsmall ++ otherchanges
+		return $ addedannexed ++ toaddsmall ++ addedpointerfiles ++ otherchanges
   where
 	(incomplete, otherchanges) = partition (\c -> isPendingAddChange c || isInProcessAddChange c) cs
 
 	returnWhen c a
 		| c = return otherchanges
 		| otherwise = a
-
+	
+	checkpointerfile change = do
+		let file = toRawFilePath $ changeFile change
+		mk <- liftIO $ isPointerFile file
+		case mk of
+			Nothing -> return (Right change)
+			Just key -> do
+				mode <- liftIO $ catchMaybeIO $ fileMode <$> R.getFileStatus file
+				liftAnnex $ stagePointerFile file mode =<< hashPointerFile key
+				return $ Left $ Change
+					(changeTime change)
+					(changeFile change)
+					(LinkChange (Just key))
+	
 	checksmall change
 		| not annexdotfiles && dotfile f =
 			return (Right change)
diff --git a/Assistant/Threads/Watcher.hs b/Assistant/Threads/Watcher.hs
index 2df29ce76c..3a72901087 100644
--- a/Assistant/Threads/Watcher.hs
+++ b/Assistant/Threads/Watcher.hs
@@ -196,11 +196,8 @@ shouldRestage :: DaemonStatus -> Bool
 shouldRestage ds = scanComplete ds || forceRestage ds
 
 onAddFile :: Bool -> Handler
-onAddFile symlinkssupported f fs = do
-	mk <- liftIO $ isPointerFile $ toRawFilePath f
-	case mk of
-		Nothing -> onAddFile' contentchanged addassociatedfile addlink samefilestatus symlinkssupported f fs
-		Just k -> addlink f k
+onAddFile symlinkssupported f fs =
+	onAddFile' contentchanged addassociatedfile addlink samefilestatus symlinkssupported f fs
   where
 	addassociatedfile key file = 
 		Database.Keys.addAssociatedFile key
diff --git a/CHANGELOG b/CHANGELOG
index fa9509fe61..9e467cfb12 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -1,3 +1,10 @@
+git-annex (10.20240702) UNRELEASED; urgency=medium
+
+  * assistant: Fix a race condition that could cause a pointer file to
+    get ingested into the annex.
+
+ -- Joey Hess <id@joeyh.name>  Tue, 02 Jul 2024 12:14:53 -0400
+
 git-annex (10.20240701) upstream; urgency=medium
 
   * git-annex remotes can now act as proxies that provide access to
diff --git a/doc/bugs/assistant___40__webapp__41___commited_unlocked_link_to_annex.mdwn b/doc/bugs/assistant___40__webapp__41___commited_unlocked_link_to_annex.mdwn
index 1b084030c1..4c14d1b40f 100644
--- a/doc/bugs/assistant___40__webapp__41___commited_unlocked_link_to_annex.mdwn
+++ b/doc/bugs/assistant___40__webapp__41___commited_unlocked_link_to_annex.mdwn
@@ -104,3 +104,6 @@ on laptop where I dive into inception: 10.20240129
 
 [[!meta author=yoh]]
 [[!tag projects/repronim]]
+
+[[!meta title="inception: pointer file can be ingested into the annex due to assistant bug or manually"]]
+
diff --git a/doc/bugs/assistant___40__webapp__41___commited_unlocked_link_to_annex/comment_7_44ab02fd027efc56ce8bb701e5513b1d._comment b/doc/bugs/assistant___40__webapp__41___commited_unlocked_link_to_annex/comment_7_44ab02fd027efc56ce8bb701e5513b1d._comment
new file mode 100644
index 0000000000..3fa5b92a77
--- /dev/null
+++ b/doc/bugs/assistant___40__webapp__41___commited_unlocked_link_to_annex/comment_7_44ab02fd027efc56ce8bb701e5513b1d._comment
@@ -0,0 +1,15 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 7"""
+ date="2024-07-02T16:15:28Z"
+ content="""
+I've fixed this race in the assistant.
+
+Question now is, can this bug be closed, or does it need to be left open,
+and git-annex made to recover from this situation? Given the complexity
+of making git-annex notice this, I'm sort of inclined to not have it
+auto-recover. Manual recovery seems pretty simple, just delete the file and
+re-add it with the right key. 
+
+Thoughts?
+"""]]

Changed pages:

bugs/assistant (webapp) commited unlocked link to annex/comment 6 3d326e7b7f6d44f5907cc808cfadee3a
Changed by: joeyh
Commit type: git
Date: 14:38:48 07/02/24

reproduced bug

diff --git a/doc/bugs/assistant___40__webapp__41___commited_unlocked_link_to_annex/comment_6_3d326e7b7f6d44f5907cc808cfadee3a._comment b/doc/bugs/assistant___40__webapp__41___commited_unlocked_link_to_annex/comment_6_3d326e7b7f6d44f5907cc808cfadee3a._comment
new file mode 100644
index 0000000000..ae87ff6036
--- /dev/null
+++ b/doc/bugs/assistant___40__webapp__41___commited_unlocked_link_to_annex/comment_6_3d326e7b7f6d44f5907cc808cfadee3a._comment
@@ -0,0 +1,25 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 6"""
+ date="2024-07-02T14:33:03Z"
+ content="""
+Added a 60 second sleep right after the assistant checks isPointerFile,
+then started the assistant and ran:
+
+	touch new
+	sleep 10
+	echo '/annex/objects/SHA256E-s30--93c16dbf65b7b66e479bd484398c09c920338e4a1df1fe352b245078d04645f4' > new
+
+Result was 2 commits, first:
+
+	+/annex/objects/SHA256E-s93--574defb13589618ec26c2516f9f62a5a1353cbea41d619034e60a697f16bd921
+
+Followed by:
+
+	-/annex/objects/SHA256E-s93--574defb13589618ec26c2516f9f62a5a1353cbea41d619034e60a697f16bd921
+	+/annex/objects/SHA256E-s30--93c16dbf65b7b66e479bd484398c09c920338e4a1df1fe352b245078d04645f4
+
+574def is the sha256sum of the annex link that I wrote to the file. So this
+does replicate the bug. Although it's odd that it then put back the annex
+link in the subsequent commit.
+"""]]

Changed pages:

todo/git-annex proxies
Changed by: joeyh
Commit type: git
Date: 15:44:54 07/01/24

reorder

diff --git a/doc/todo/git-annex_proxies.mdwn b/doc/todo/git-annex_proxies.mdwn
index d5430a60bb..63366bfee6 100644
--- a/doc/todo/git-annex_proxies.mdwn
+++ b/doc/todo/git-annex_proxies.mdwn
@@ -18,8 +18,8 @@ Joey has received funding to work on this.
 Planned schedule of work:
 
 * June: git-annex proxies and clusters
-* July, part 1: git-annex proxy support for exporttree
-* July, part 2: p2p protocol over http
+* July, part 1: p2p protocol over http
+* July, part 2: git-annex proxy support for exporttree
 * August: balanced preferred content
 * September: streaming through proxy to special remotes (especially S3)
 * October: proving behavior of balanced preferred content with proxies

Changed pages:

todo/git-annex proxies
Changed by: joeyh
Commit type: git
Date: 15:38:29 07/01/24

update

diff --git a/doc/todo/git-annex_proxies.mdwn b/doc/todo/git-annex_proxies.mdwn
index a93a5d4dfe..d5430a60bb 100644
--- a/doc/todo/git-annex_proxies.mdwn
+++ b/doc/todo/git-annex_proxies.mdwn
@@ -8,6 +8,12 @@ repositories.
 4. [[todo/track_free_space_in_repos_via_git-annex_branch]]
 5. [[todo/proving_preferred_content_behavior]]
 
+## table of contents
+
+[[!toc ]]
+
+## planned schedule
+
 Joey has received funding to work on this.
 Planned schedule of work:
 
@@ -20,9 +26,7 @@ Planned schedule of work:
 
 [[!tag projects/openneuro]]
 
-[[!toc ]]
-
-# work notes
+## work notes
 
 For June's work on [[design/passthrough_proxy]], remaining todos:
 
@@ -30,7 +34,7 @@ For June's work on [[design/passthrough_proxy]], remaining todos:
   to avoid the proxy's disk filling up with the temporary object file
   cached there.
 
-# items deferred until later for [[design/passthrough_proxy]]
+## items deferred until later for [[design/passthrough_proxy]]
 
 * Resuming an interrupted download from proxied special remote makes the proxy
   re-download the whole content. It could instead keep some of the 
@@ -78,7 +82,7 @@ For June's work on [[design/passthrough_proxy]], remaining todos:
 * Support using a proxy when its url is a P2P address.
   (Eg tor-annex remotes.)
 
-# completed items for June's work on [[design/passthrough_proxy]]:
+## completed items for June's work on [[design/passthrough_proxy]]:
 
 * UUID discovery via git-annex branch. Add a log file listing UUIDs
   accessible via proxy UUIDs. It also will contain the names

Changed pages:

todo/git-annex proxies
Changed by: joeyh
Commit type: git
Date: 15:37:12 07/01/24

toc

diff --git a/doc/todo/git-annex_proxies.mdwn b/doc/todo/git-annex_proxies.mdwn
index c385bd48cd..a93a5d4dfe 100644
--- a/doc/todo/git-annex_proxies.mdwn
+++ b/doc/todo/git-annex_proxies.mdwn
@@ -20,6 +20,8 @@ Planned schedule of work:
 
 [[!tag projects/openneuro]]
 
+[[!toc ]]
+
 # work notes
 
 For June's work on [[design/passthrough_proxy]], remaining todos:

Changed pages:

git-annex

tips/clusters
Changed by: joeyh
Commit type: git
Date: 15:33:55 07/01/24

document proxying to special remotes

diff --git a/doc/git-annex.mdwn b/doc/git-annex.mdwn
index 3d95cd5b1a..51438ef7d9 100644
--- a/doc/git-annex.mdwn
+++ b/doc/git-annex.mdwn
@@ -1677,7 +1677,7 @@ Remotes are configured using these settings in `.git/config`.
 * `remote.<name>.annex-proxy`
 
   Set to "true" to make the local repository able to act as a proxy to this
-  remote. 
+  remote. The remote can be a git-annex repository or a special remote.
 
   After configuring this, run [[git-annex-updateproxy](1) to store
   the new configuration in the git-annex branch.
diff --git a/doc/tips/clusters.mdwn b/doc/tips/clusters.mdwn
index 320e77f9a5..e9fc4d5bcc 100644
--- a/doc/tips/clusters.mdwn
+++ b/doc/tips/clusters.mdwn
@@ -4,6 +4,9 @@ form a single logical repository.
 A cluster is accessed via a gateway repository. The gateway is not itself
 a node of the cluster.
 
+A cluster's nodes can be any combination of git-annex repositories and
+special remotes.
+
 [[!toc ]]
 
 ## using a cluster

Changed pages:

todo/git-annex proxies
Changed by: joeyh
Commit type: git
Date: 15:33:07 07/01/24

update

diff --git a/doc/todo/git-annex_proxies.mdwn b/doc/todo/git-annex_proxies.mdwn
index d1a94b2717..c385bd48cd 100644
--- a/doc/todo/git-annex_proxies.mdwn
+++ b/doc/todo/git-annex_proxies.mdwn
@@ -40,6 +40,12 @@ For June's work on [[design/passthrough_proxy]], remaining todos:
 * Streaming download from proxied special remotes. See design.
   (Planned for September)
 
+* When an upload to a cluster is distributed to multiple special remotes,
+  a temporary file is written for each one, which may even happen in
+  parallel. This is a lot of extra work and may use excess disk space.
+  It should be possible to only write a single temp file.
+  (With streaming this won't be an issue.)
+
 * Indirect uploads when proxying for special remote
   (to be considered). See design.

Changed pages:

todo/git-annex proxies
Changed by: joeyh
Commit type: git
Date: 15:29:04 07/01/24

update

diff --git a/doc/todo/git-annex_proxies.mdwn b/doc/todo/git-annex_proxies.mdwn
index ed27702122..d1a94b2717 100644
--- a/doc/todo/git-annex_proxies.mdwn
+++ b/doc/todo/git-annex_proxies.mdwn
@@ -24,6 +24,12 @@ Planned schedule of work:
 
 For June's work on [[design/passthrough_proxy]], remaining todos:
 
+* Check annex.diskreserve when proxying for special remotes
+  to avoid the proxy's disk filling up with the temporary object file
+  cached there.
+
+# items deferred until later for [[design/passthrough_proxy]]
+
 * Resuming an interrupted download from proxied special remote makes the proxy
   re-download the whole content. It could instead keep some of the 
   object files around when the client does not send SUCCESS. This would
@@ -32,10 +38,7 @@ For June's work on [[design/passthrough_proxy]], remaining todos:
   The design doc has some more thoughts about this.
 
 * Streaming download from proxied special remotes. See design.
-
-* Check annex.diskreserve when proxying for special remotes.
-
-# items deferred until later for [[design/passthrough_proxy]]
+  (Planned for September)
 
 * Indirect uploads when proxying for special remote
   (to be considered). See design.
@@ -130,3 +133,7 @@ For June's work on [[design/passthrough_proxy]], remaining todos:
 
 * Basic support for proxying special remotes. (But not exporttree=yes ones
   yet.) (done)
+
+* Tab complete remotes in all relevant commands (done)
+
+* Display cluster and proxy information in git-annex info (done)

Changed pages:

todo/git-annex proxies
Changed by: joeyh
Commit type: git
Date: 15:19:02 07/01/24

fix display when proxied GET yields ERROR
The error message is not displayed to the use, but this mirrors the
behavior when a regular get from a special remote fails. At least now
there is not a protocol error.

diff --git a/Annex/Proxy.hs b/Annex/Proxy.hs
index 50b108bab4..167a76ca47 100644
--- a/Annex/Proxy.hs
+++ b/Annex/Proxy.hs
@@ -114,7 +114,7 @@ proxySpecialRemote protoversion r ihdl ohdl owaitv endv = go
 			liftIO $ sendmessage $
 				ERROR "NOTIFYCHANGE unsupported for a special remote"
 			go
-		Just _ -> giveup "protocol error"
+		Just _ -> giveup "protocol error M"
 		Nothing -> return ()
 
 	getnextmessageorend = 
@@ -167,9 +167,9 @@ proxySpecialRemote protoversion r ihdl ohdl owaitv endv = go
 								store
 							Just (VALIDITY Invalid) ->
 								return ()
-							_ -> giveup "protocol error"
+							_ -> giveup "protocol error N"
 						else store
-				_ -> giveup "protocol error"
+				_ -> giveup "protocol error O"
 
 	proxyget offset af k = withproxytmpfile k $ \tmpfile -> do
 		-- Don't verify the content from the remote,
@@ -206,6 +206,6 @@ proxySpecialRemote protoversion r ihdl ohdl owaitv endv = go
 			receivemessage >>= \case
 				Just SUCCESS -> return ()
 				Just FAILURE -> return ()
-				Just _ -> giveup "protocol error"
+				Just _ -> giveup "protocol error P"
 				Nothing -> return ()
 						
diff --git a/P2P/IO.hs b/P2P/IO.hs
index 42b53a671a..14c588dac5 100644
--- a/P2P/IO.hs
+++ b/P2P/IO.hs
@@ -216,7 +216,7 @@ runNet runst conn runner f = case f of
 			Right () -> runner next
 	ReceiveMessage next ->
 		let protoerr = return $ Left $
-			ProtoFailureMessage "protocol error"
+			ProtoFailureMessage "protocol error 1"
 		    gotmessage m = do
 			liftIO $ debugMessage conn "P2P <" m
 			runner (next (Just m))
@@ -263,7 +263,7 @@ runNet runst conn runner f = case f of
 				liftIO (atomically (takeTMVar mv)) >>= \case
 					Left b -> runner (next b)
 					Right _ -> return $ Left $
-						ProtoFailureMessage "protocol error"
+						ProtoFailureMessage "protocol error 2"
 	CheckAuthToken _u t next -> do
 		let authed = connCheckAuth conn t
 		runner (next authed)
diff --git a/P2P/Protocol.hs b/P2P/Protocol.hs
index 1a3bcd5d7e..79d8fbd8a3 100644
--- a/P2P/Protocol.hs
+++ b/P2P/Protocol.hs
@@ -623,6 +623,8 @@ receiveContent mm p sizer storer mkmsg = do
 				validitycheck
 			sendSuccess (observeBool v)
 			return v
+		Just (ERROR _err) ->
+			return observeFailure
 		_ -> do
 			net $ sendMessage (ERROR "expected DATA")
 			return observeFailure
diff --git a/P2P/Proxy.hs b/P2P/Proxy.hs
index 45085f4734..53cc4ff947 100644
--- a/P2P/Proxy.hs
+++ b/P2P/Proxy.hs
@@ -315,8 +315,8 @@ proxy proxydone proxymethods servermode (ClientSide clientrunst clientconn) remo
 					to $ net $ sendMessage message
 	
 	protoerr = do
-		_ <- client $ net $ sendMessage (ERROR "protocol error")
-		giveup "protocol error"
+		_ <- client $ net $ sendMessage (ERROR "protocol error X")
+		giveup "protocol error M"
 	
 	handleREMOVE [] _ _ =
 		-- When no places are provided to remove from,
@@ -357,7 +357,10 @@ proxy proxydone proxymethods servermode (ClientSide clientrunst clientconn) remo
 						else FAILURE_PLUS us
 
 	handleGET remoteside message = getresponse (runRemoteSide remoteside) message $
-		withDATA (relayGET remoteside)
+		withDATA (relayGET remoteside) $ \case
+			ERROR err -> protoerrhandler proxynextclientmessage $
+				client $ net $ sendMessage (ERROR err)
+			_ -> protoerr
 
 	handlePUT (remoteside:[]) k message
 		| Remote.uuid (remote remoteside) == remoteuuid =
@@ -368,7 +371,9 @@ proxy proxydone proxymethods servermode (ClientSide clientrunst clientconn) remo
 					client $ net $ sendMessage resp
 				PUT_FROM _ -> 
 					getresponse client resp $ 
-						withDATA (relayPUT remoteside k)
+						withDATA
+							(relayPUT remoteside k)
+							(const protoerr)
 				_ -> protoerr
 	handlePUT [] _ _ = 
 		protoerrhandler proxynextclientmessage $
@@ -376,8 +381,8 @@ proxy proxydone proxymethods servermode (ClientSide clientrunst clientconn) remo
 	handlePUT remotesides k message = 
 		handlePutMulti remotesides k message
 
-	withDATA a message@(DATA len) = a len message
-	withDATA _ _ = protoerr
+	withDATA a _ message@(DATA len) = a len message
+	withDATA _ a message = a message
 	
 	relayGET remoteside len = relayDATAStart client $
 		relayDATACore len (runRemoteSide remoteside) client $
@@ -438,7 +443,8 @@ proxy proxydone proxymethods servermode (ClientSide clientrunst clientconn) remo
 					let l' = rights (rights l)
 					let minoffset = minimum (map snd l')
 					getresponse client (PUT_FROM (Offset minoffset)) $
-						withDATA (relayPUTMulti minoffset l' k)	
+						withDATA (relayPUTMulti minoffset l' k)
+							(const protoerr)
 	
 	relayPUTMulti minoffset remotes k (Len datalen) _ = do
 		let totallen = datalen + minoffset
diff --git a/doc/todo/git-annex_proxies.mdwn b/doc/todo/git-annex_proxies.mdwn
index fb8ece8da6..ed27702122 100644
--- a/doc/todo/git-annex_proxies.mdwn
+++ b/doc/todo/git-annex_proxies.mdwn
@@ -31,9 +31,6 @@ For June's work on [[design/passthrough_proxy]], remaining todos:
   needs some disk. And it could minimize to eg, the last 2 or so.
   The design doc has some more thoughts about this.
 
-* If GET from a proxied special remote sends an ERROR with a message
-  from the special remote, currently the user sees "protocol error".
-
 * Streaming download from proxied special remotes. See design.
 
 * Check annex.diskreserve when proxying for special remotes.

Changed pages:

todo/git-annex proxies
Changed by: joeyh
Commit type: git
Date: 14:53:49 07/01/24

avoid populating proxy's object file when storing on special remote
Now that storeKey can have a different object file passed to it, this
complication is not needed. This avoids a lot of strange situations,
and will also be needed if streaming is eventually supported.

diff --git a/Annex/Proxy.hs b/Annex/Proxy.hs
index ccd0057611..50b108bab4 100644
--- a/Annex/Proxy.hs
+++ b/Annex/Proxy.hs
@@ -149,57 +149,27 @@ proxySpecialRemote protoversion r ihdl ohdl owaitv endv = go
 			a (toRawFilePath tmpdir P.</> keyFile k)
 			
 	proxyput af k = do
-		-- In order to send to the special remote, the key will
-		-- need to be inserted into the object directory.
-		-- It will be dropped again afterwards. Unless it's already
-		-- present there.
-		ifM (inAnnex k)
-			( tryNonAsync (Remote.storeKey r k af Nothing nullMeterUpdate) >>= \case
-				Right () -> liftIO $ sendmessage ALREADY_HAVE
+		liftIO $ sendmessage $ PUT_FROM (Offset 0)
+		withproxytmpfile k $ \tmpfile -> do
+			let store = tryNonAsync (Remote.storeKey r k af (Just (decodeBS tmpfile)) nullMeterUpdate) >>= \case
+				Right () -> liftIO $ sendmessage SUCCESS
 				Left err -> liftIO $ propagateerror err
-			, do
-				liftIO $ sendmessage $ PUT_FROM (Offset 0)
-				ifM receivedata
-					( do
-						tryNonAsync (Remote.storeKey r k af Nothing nullMeterUpdate) >>= \case
-							Right () -> do
-								depopulateobjectfile
-								liftIO $ sendmessage SUCCESS
-							Left err -> do
-								depopulateobjectfile
-								liftIO $ propagateerror err
-					, liftIO $ sendmessage FAILURE
-					)
-			)
-	  where
-		receivedata = withproxytmpfile k $ \tmpfile ->
 			liftIO receivemessage >>= \case
 				Just (DATA (Len _)) -> do
 					b <- liftIO receivebytestring
 					liftIO $ L.writeFile (fromRawFilePath tmpfile) b
 					-- Signal that the whole bytestring
-					-- has been stored.
+					-- has been received.
 					liftIO $ atomically $ putTMVar owaitv ()
 					if protoversion > ProtocolVersion 1
-						then do
-							liftIO receivemessage >>= \case
-								Just (VALIDITY Valid) ->
-									populateobjectfile tmpfile
-								Just (VALIDITY Invalid) -> return False
-								_ -> giveup "protocol error"
-						else populateobjectfile tmpfile
+						then liftIO receivemessage >>= \case
+							Just (VALIDITY Valid) ->
+								store
+							Just (VALIDITY Invalid) ->
+								return ()
+							_ -> giveup "protocol error"
+						else store
 				_ -> giveup "protocol error"
-					
-		populateobjectfile tmpfile = 
-			getViaTmpFromDisk Remote.RetrievalAllKeysSecure Remote.DefaultVerify k af $ \dest -> do
-				unVerified $ do
-					liftIO $ renameFile
-						(fromRawFilePath tmpfile)
-						(fromRawFilePath dest)
-					return True
-						
-		depopulateobjectfile = void $ tryNonAsync $ 
-			lockContentForRemoval k noop removeAnnex
 
 	proxyget offset af k = withproxytmpfile k $ \tmpfile -> do
 		-- Don't verify the content from the remote,
diff --git a/doc/todo/git-annex_proxies.mdwn b/doc/todo/git-annex_proxies.mdwn
index 8d803ec893..fb8ece8da6 100644
--- a/doc/todo/git-annex_proxies.mdwn
+++ b/doc/todo/git-annex_proxies.mdwn
@@ -34,17 +34,6 @@ For June's work on [[design/passthrough_proxy]], remaining todos:
 * If GET from a proxied special remote sends an ERROR with a message
   from the special remote, currently the user sees "protocol error".
 
-* convert Remote.storeKey to take the path of the object file to send.
-  It's too ugly that PUT to a proxied special remote currently has to
-  temporarily populate the proxy's annex object file. There are too many
-  ways that could lead to surprising behavior, like an interrupted PUT
-  leaving it populated, or simulantaneous PUTs.
-
-* PUT to a proxied special remote, in the case where the proxy contains
-  the key, and the special remote is not accessible, sends back ERROR
-  rather than PUT-FROM or ALREADY-HAVE. Verify that the client processes
-  that ok and displays it to the user.
-
 * Streaming download from proxied special remotes. See design.
 
 * Check annex.diskreserve when proxying for special remotes.

Changed pages:

todo/git-annex proxies
Changed by: joeyh
Commit type: git
Date: 14:04:45 07/01/24

dup stdio handles for P2P proxy
Special remotes might output to stdout, or read from stdin, which would
mess up the P2P protocol. So dup the handles to avoid any such problem.

diff --git a/Command/P2PStdIO.hs b/Command/P2PStdIO.hs
index 4b38057e58..910d9cb7cc 100644
--- a/Command/P2PStdIO.hs
+++ b/Command/P2PStdIO.hs
@@ -99,7 +99,7 @@ performProxyCluster clientuuid clusteruuid servermode = do
 proxyClientSide :: UUID -> Annex ClientSide
 proxyClientSide clientuuid = do
 	clientrunst <- liftIO (mkRunState $ Serving clientuuid Nothing)
-	return $ ClientSide clientrunst (stdioP2PConnection Nothing)
+	ClientSide clientrunst <$> liftIO (stdioP2PConnectionDupped Nothing)
 
 p2pErrHandler :: Annex () -> (a -> CommandPerform) -> Annex (Either ProtoFailure a) -> CommandPerform
 p2pErrHandler closeconn cont a = a >>= \case
diff --git a/P2P/IO.hs b/P2P/IO.hs
index 15e0dccde4..42b53a671a 100644
--- a/P2P/IO.hs
+++ b/P2P/IO.hs
@@ -1,6 +1,6 @@
 {- P2P protocol, IO implementation
  -
- - Copyright 2016-2018 Joey Hess <id@joeyh.name>
+ - Copyright 2016-2024 Joey Hess <id@joeyh.name>
  -
  - Licensed under the GNU AGPL version 3 or higher.
  -}
@@ -16,6 +16,7 @@ module P2P.IO
 	, ConnIdent(..)
 	, ClosableConnection(..)
 	, stdioP2PConnection
+	, stdioP2PConnectionDupped
 	, connectPeer
 	, closeConnection
 	, serveUnixSocket
@@ -104,6 +105,20 @@ stdioP2PConnection g = P2PConnection
 	, connIdent = ConnIdent Nothing
 	}
 
+-- P2PConnection using stdio, but with the handles first duplicated,
+-- to avoid anything that might output to stdio (eg a program run by a
+-- special remote) from interfering with the connection.
+stdioP2PConnectionDupped :: Maybe Git.Repo -> IO P2PConnection
+stdioP2PConnectionDupped g = do
+	(readh, writeh) <- dupIoHandles
+	return $ P2PConnection
+		{ connRepo = g
+		, connCheckAuth = const False
+		, connIhdl = P2PHandle readh
+		, connOhdl = P2PHandle writeh
+		, connIdent = ConnIdent Nothing
+		}
+
 -- Opens a connection to a peer. Does not authenticate with it.
 connectPeer :: Maybe Git.Repo -> P2PAddress -> IO P2PConnection
 connectPeer g (TorAnnex onionaddress onionport) = do
diff --git a/doc/todo/git-annex_proxies.mdwn b/doc/todo/git-annex_proxies.mdwn
index 50eb5aeef3..8d803ec893 100644
--- a/doc/todo/git-annex_proxies.mdwn
+++ b/doc/todo/git-annex_proxies.mdwn
@@ -45,13 +45,6 @@ For June's work on [[design/passthrough_proxy]], remaining todos:
   rather than PUT-FROM or ALREADY-HAVE. Verify that the client processes
   that ok and displays it to the user.
 
-* If a special remote outputs to stdout, or reads from stdin, that will
-  mess up the P2P protocol. Move the special remote proxying into a
-  separate process perhaps, which can be run with stdout and stdin
-  redirected? Or, fix any special remotes that might do that. Are
-  there any left? External special remotes certainly don't since that would
-  mess up their own protocol. Hook special remotes can though.
-
 * Streaming download from proxied special remotes. See design.
 
 * Check annex.diskreserve when proxying for special remotes.

Changed pages:

bugs/keeps trying to commit file unlocked
Changed by: yarikoptic
Commit type: web
Date: 18:52:23 06/30/24

initial report on file jumping from locked to unlocked

diff --git a/doc/bugs/keeps_trying_to_commit_file_unlocked.mdwn b/doc/bugs/keeps_trying_to_commit_file_unlocked.mdwn
new file mode 100644
index 0000000000..d584c9d11c
--- /dev/null
+++ b/doc/bugs/keeps_trying_to_commit_file_unlocked.mdwn
@@ -0,0 +1,92 @@
+### Please describe the problem.
+
+I do not remember trying to add that file "unlocked" anyhow, but git-annex, upon 2nd invocation of `git status` flips the file claiming it should get to the "unlocked" git link from original symlink:
+
+```
+❯ git annex version
+git-annex version: 10.20240531+git214-g28f5c47b5a-1~ndall+1
+
+❯ git status
+On branch master
+Your branch is ahead of 'origin/master' by 1 commit.
+  (use "git push" to publish your local commits)
+
+Changes not staged for commit:
+  (use "git add <file>..." to update what will be committed)
+  (use "git restore <file>..." to discard changes in working directory)
+	modified:   samples/UK_gla_3T_fMRI_consent_form_v3.0.docx
+
+no changes added to commit (use "git add" and/or "git commit -a")
+
+❯ git reset --hard
+HEAD is now at 8700315 BF: fix leftover merge conflict in CONTRIBUTING.rst
+```
+
+so here from a clean state on first `git status` we get to dirty on subsequent:
+
+
+```
+❯ git status
+git-annex: git status will show samples/UK_gla_3T_fMRI_consent_form_v3.0.docx to be modified, since content availability has changed and git-annex was unable to update the index. This is only a cosmetic problem affecting git status; git add, git commit, etc won't be affected. To fix the git status display, you can run: git-annex restage
+On branch master
+Your branch is ahead of 'origin/master' by 1 commit.
+  (use "git push" to publish your local commits)
+
+nothing to commit, working tree clean
+
+❯ git status
+On branch master
+Your branch is ahead of 'origin/master' by 1 commit.
+  (use "git push" to publish your local commits)
+
+Changes not staged for commit:
+  (use "git add <file>..." to update what will be committed)
+  (use "git restore <file>..." to discard changes in working directory)
+	modified:   samples/UK_gla_3T_fMRI_consent_form_v3.0.docx
+
+no changes added to commit (use "git add" and/or "git commit -a")
+
+❯ git diff
+diff --git a/samples/UK_gla_3T_fMRI_consent_form_v3.0.docx b/samples/UK_gla_3T_fMRI_consent_form_v3.0.docx
+index 3215574..488e63f 100644
+--- a/samples/UK_gla_3T_fMRI_consent_form_v3.0.docx
++++ b/samples/UK_gla_3T_fMRI_consent_form_v3.0.docx
+@@ -1 +1 @@
+-../.git/annex/objects/5M/wv/MD5E-s591826--1ca9251906259623f73a3aba47ef6369.0.docx/MD5E-s591826--1ca9251906259623f73a3aba47ef6369.0.docx
+\ No newline at end of file
++/annex/objects/MD5E-s591826--1ca9251906259623f73a3aba47ef6369.0.docx
+
+❯ git annex restage
+restage  ok
+
+❯ git diff
+
+❯ git status
+On branch master
+Your branch is ahead of 'origin/master' by 1 commit.
+  (use "git push" to publish your local commits)
+
+Changes to be committed:
+  (use "git restore --staged <file>..." to unstage)
+	modified:   samples/UK_gla_3T_fMRI_consent_form_v3.0.docx
+
+❯ git diff --cached
+diff --git a/samples/UK_gla_3T_fMRI_consent_form_v3.0.docx b/samples/UK_gla_3T_fMRI_consent_form_v3.0.docx
+index 3215574..488e63f 100644
+--- a/samples/UK_gla_3T_fMRI_consent_form_v3.0.docx
++++ b/samples/UK_gla_3T_fMRI_consent_form_v3.0.docx
+@@ -1 +1 @@
+-../.git/annex/objects/5M/wv/MD5E-s591826--1ca9251906259623f73a3aba47ef6369.0.docx/MD5E-s591826--1ca9251906259623f73a3aba47ef6369.0.docx
+\ No newline at end of file
++/annex/objects/MD5E-s591826--1ca9251906259623f73a3aba47ef6369.0.docx
+
+```
+
+This is on a local long-lived clone of http://github.com/con/open-brain-consent but seems to not happen on a fresh clone. So must be something about local "state".
+
+Not sure if relates to the following issue under current investigation (but not yet addressed):
+
+- [https://git-annex.branchable.com/bugs/assistant___40__webapp__41___commited_unlocked_link_to_annex/](https://git-annex.branchable.com/bugs/assistant___40__webapp__41___commited_unlocked_link_to_annex/)
+
+[[!meta author=yoh]]
+[[!tag projects/repronim]]

Changed pages:

bugs/change in beh: addurls creates multiple commits
Changed by: yarikoptic
Commit type: web
Date: 16:04:30 06/30/24

original report on change in behavior with addurl --batch

diff --git a/doc/bugs/change_in_beh__58___addurls_creates_multiple_commits.mdwn b/doc/bugs/change_in_beh__58___addurls_creates_multiple_commits.mdwn
new file mode 100644
index 0000000000..2b9d5c5905
--- /dev/null
+++ b/doc/bugs/change_in_beh__58___addurls_creates_multiple_commits.mdwn
@@ -0,0 +1,188 @@
+### Please describe the problem.
+
+Our DataLad test which explicitly tests that we are not breeding commits in git-annex branch while adding files/urls to point to datalad-archive special remote started to fail going from git-annex 10.20240532-gf9ce7a452cc0fd5cdd2d58739741f7264fdbc598 to 10.20240532-g28f5c47b5a0daf96e5ed9aa719ff1e2763d3cc8b
+(invocation: `python -m pytest -s -v datalad/local/tests/test_add_archive_content.py::TestAddArchiveOptions::test_add_delete_after_and_drop_subdir`)
+
+If before we had a single commit
+<details>
+<summary></summary> 
+
+```shell
+❯ git log -p git-annex^..git-annex
+commit b42433cab9f671d206fe937ee7b68b53f11a0c54 (git-annex)
+Author: DataLad Tester <test@example.com>
+Date:   Sun Jun 30 10:48:16 2024 -0400
+
+    update
+
+diff --git a/d77/a0b/MD5E-s4--ec4d1eb36b22d19728e9d1d23ca84d1c.txt.log b/d77/a0b/MD5E-s4--ec4d1eb36b22d19728e9d1d23ca84d1c.txt.log
+new file mode 100644
+index 0000000..cc638db
+--- /dev/null
++++ b/d77/a0b/MD5E-s4--ec4d1eb36b22d19728e9d1d23ca84d1c.txt.log
+@@ -0,0 +1,2 @@
++1719758896s 1 c04eb54b-4b4e-5755-8436-866b043170fa
++1719758897s 0 d53ab0e3-21a9-4084-806f-bf9f5812f34e
+diff --git a/d77/a0b/MD5E-s4--ec4d1eb36b22d19728e9d1d23ca84d1c.txt.log.web b/d77/a0b/MD5E-s4--ec4d1eb36b22d19728e9d1d23ca84d1c.txt.log.web
+new file mode 100644
+index 0000000..8ef0f1f
+--- /dev/null
++++ b/d77/a0b/MD5E-s4--ec4d1eb36b22d19728e9d1d23ca84d1c.txt.log.web
+@@ -0,0 +1 @@
++1719758896s 1 :dl+archive:MD5E-s3584--2f350c3650d5e3a21785d55f5a94ce70.tar#path=1/file.txt&size=4
+diff --git a/f45/7f1/MD5E-s5--db87ebcba59a8c9f34b68e713c08a718.dat.log b/f45/7f1/MD5E-s5--db87ebcba59a8c9f34b68e713c08a718.dat.log
+new file mode 100644
+index 0000000..cc638db
+--- /dev/null
++++ b/f45/7f1/MD5E-s5--db87ebcba59a8c9f34b68e713c08a718.dat.log
+@@ -0,0 +1,2 @@
++1719758896s 1 c04eb54b-4b4e-5755-8436-866b043170fa
++1719758897s 0 d53ab0e3-21a9-4084-806f-bf9f5812f34e
+diff --git a/f45/7f1/MD5E-s5--db87ebcba59a8c9f34b68e713c08a718.dat.log.web b/f45/7f1/MD5E-s5--db87ebcba59a8c9f34b68e713c08a718.dat.log.web
+new file mode 100644
+index 0000000..30bb5e9
+--- /dev/null
++++ b/f45/7f1/MD5E-s5--db87ebcba59a8c9f34b68e713c08a718.dat.log.web
+@@ -0,0 +1 @@
++1719758896s 1 :dl+archive:MD5E-s3584--2f350c3650d5e3a21785d55f5a94ce70.tar#path=1/1.dat&size=5
+
+```
+</details>
+
+<details>
+<summary>now we got two</summary> 
+
+```shell
+Author: DataLad Tester <test@example.com>
+Date:   Sun Jun 30 10:45:12 2024 -0400
+
+    update
+
+diff --git a/d77/a0b/MD5E-s4--ec4d1eb36b22d19728e9d1d23ca84d1c.txt.log b/d77/a0b/MD5E-s4--ec4d1eb36b22d19728e9d1d23ca84d1c.txt.log
+new file mode 100644
+index 0000000..97acf53
+--- /dev/null
++++ b/d77/a0b/MD5E-s4--ec4d1eb36b22d19728e9d1d23ca84d1c.txt.log
+@@ -0,0 +1,2 @@
++1719758713s 0 86661c7b-0604-49e7-8d65-1baf4ca9f469
++1719758712s 1 c04eb54b-4b4e-5755-8436-866b043170fa
+diff --git a/d77/a0b/MD5E-s4--ec4d1eb36b22d19728e9d1d23ca84d1c.txt.log.web b/d77/a0b/MD5E-s4--ec4d1eb36b22d19728e9d1d23ca84d1c.txt.log.web
+new file mode 100644
+index 0000000..e5bafba
+--- /dev/null
++++ b/d77/a0b/MD5E-s4--ec4d1eb36b22d19728e9d1d23ca84d1c.txt.log.web
+@@ -0,0 +1 @@
++1719758712s 1 :dl+archive:MD5E-s3584--de6498c9ca26fee011f289f5f5972ed0.tar#path=1/file.txt&size=4
+diff --git a/f45/7f1/MD5E-s5--db87ebcba59a8c9f34b68e713c08a718.dat.log b/f45/7f1/MD5E-s5--db87ebcba59a8c9f34b68e713c08a718.dat.log
+index 11934b6..97acf53 100644
+--- a/f45/7f1/MD5E-s5--db87ebcba59a8c9f34b68e713c08a718.dat.log
++++ b/f45/7f1/MD5E-s5--db87ebcba59a8c9f34b68e713c08a718.dat.log
+@@ -1,2 +1,2 @@
+-1719758712s 1 86661c7b-0604-49e7-8d65-1baf4ca9f469
++1719758713s 0 86661c7b-0604-49e7-8d65-1baf4ca9f469
+ 1719758712s 1 c04eb54b-4b4e-5755-8436-866b043170fa
+
+commit 8c4fdbadb4b1735cbb47f833ef99235790b8bcbf
+Author: DataLad Tester <test@example.com>
+Date:   Sun Jun 30 10:45:12 2024 -0400
+
+    update
+
+diff --git a/f45/7f1/MD5E-s5--db87ebcba59a8c9f34b68e713c08a718.dat.log b/f45/7f1/MD5E-s5--db87ebcba59a8c9f34b68e713c08a718.dat.log
+new file mode 100644
+index 0000000..11934b6
+--- /dev/null
++++ b/f45/7f1/MD5E-s5--db87ebcba59a8c9f34b68e713c08a718.dat.log
+@@ -0,0 +1,2 @@
++1719758712s 1 86661c7b-0604-49e7-8d65-1baf4ca9f469
++1719758712s 1 c04eb54b-4b4e-5755-8436-866b043170fa
+diff --git a/f45/7f1/MD5E-s5--db87ebcba59a8c9f34b68e713c08a718.dat.log.web b/f45/7f1/MD5E-s5--db87ebcba59a8c9f34b68e713c08a718.dat.log.web
+new file mode 100644
+index 0000000..107c66f
+--- /dev/null
++++ b/f45/7f1/MD5E-s5--db87ebcba59a8c9f34b68e713c08a718.dat.log.web
+@@ -0,0 +1 @@
++1719758712s 1 :dl+archive:MD5E-s3584--de6498c9ca26fee011f289f5f5972ed0.tar#path=1/1.dat&size=5
+```
+</details>
+
+for the same effect.  And I believe the command which triggers them is `['git', '-c', 'diff.ignoreSubmodules=none', '-c', 'core.quotepath=false', 'annex', 'addurl', '--with-files', '--json', '--json-error-messages', '--batch']` which before (for years?!) resulted in expected single commit.
+
+<details>
+<summary>Here is the full set of datalad logs for the steps triggering that </summary> 
+
+```shell
+[DEBUG  ] Determined class of decorated function: <class 'datalad.local.add_archive_content.AddArchiveContent'> 
+[DEBUG  ] Resolved dataset to add-archive-content: /home/yoh/.tmp/datalad_temp_tree_rsua9kmg 
+[DEBUG  ] Determined class of decorated function: <class 'datalad.core.local.status.Status'> 
+[DEBUG  ] Resolved dataset to report status: /home/yoh/.tmp/datalad_temp_tree_rsua9kmg 
+[DEBUG  ] Querying AnnexRepo(/home/yoh/.tmp/datalad_temp_tree_rsua9kmg).diffstatus() for paths: [PosixPath('/home/yoh/.tmp/datalad_temp_tree_rsua9kmg/subdir/1.tar')] 
+[DEBUG  ] Run ['git', '-c', 'diff.ignoreSubmodules=none', '-c', 'core.quotepath=false', 'rev-parse', '--quiet', '--verify', 'HEAD^{commit}'] (protocol_class=GeneratorStdOutErrCapture) (cwd=/home/yoh/.tmp/datalad_temp_tree_rsua9kmg) 
+[DEBUG  ] AnnexRepo(/home/yoh/.tmp/datalad_temp_tree_rsua9kmg).get_content_info(...) 
+[DEBUG  ] Query repo: ['ls-files', '--stage', '-z', '--exclude-standard', '-o', '--directory', '--no-empty-directory'] 
+[DEBUG  ] Run ['git', '-c', 'diff.ignoreSubmodules=none', '-c', 'core.quotepath=false', 'ls-files', '--stage', '-z', '--exclude-standard', '-o', '--directory', '--no-empty-directory', '--', 'subdir/1.tar'] (protocol_class=GeneratorStdOutErrCapture) (cwd=/home/yoh/.tmp/datalad_temp_tree_rsua9kmg) 
+[DEBUG  ] Done query repo: ['ls-files', '--stage', '-z', '--exclude-standard', '-o', '--directory', '--no-empty-directory'] 
+[DEBUG  ] Done AnnexRepo(/home/yoh/.tmp/datalad_temp_tree_rsua9kmg).get_content_info(...) 
+[DEBUG  ] Run ['git', '-c', 'diff.ignoreSubmodules=none', '-c', 'core.quotepath=false', 'ls-files', '-z', '-m', '-d', '--', 'subdir/1.tar'] (protocol_class=GeneratorStdOutErrCapture) (cwd=/home/yoh/.tmp/datalad_temp_tree_rsua9kmg) 
+[DEBUG  ] AnnexRepo(/home/yoh/.tmp/datalad_temp_tree_rsua9kmg).get_content_info(...) 
+[DEBUG  ] Query repo: ['ls-tree', 'HEAD', '-z', '-r', '--full-tree', '-l'] 
+[DEBUG  ] Run ['git', '-c', 'diff.ignoreSubmodules=none', '-c', 'core.quotepath=false', 'ls-tree', 'HEAD', '-z', '-r', '--full-tree', '-l', '--', 'subdir/1.tar'] (protocol_class=GeneratorStdOutErrCapture) (cwd=/home/yoh/.tmp/datalad_temp_tree_rsua9kmg) 
+[DEBUG  ] Done query repo: ['ls-tree', 'HEAD', '-z', '-r', '--full-tree', '-l'] 
+[DEBUG  ] Done AnnexRepo(/home/yoh/.tmp/datalad_temp_tree_rsua9kmg).get_content_info(...) 
+[DEBUG  ] Run ['git', '-c', 'diff.ignoreSubmodules=none', '-c', 'core.quotepath=false', 'status', '--porcelain', '--untracked-files=normal', '--ignore-submodules=none'] (protocol_class=GeneratorStdOutErrCapture) (cwd=/home/yoh/.tmp/datalad_temp_tree_rsua9kmg) 
+[DEBUG  ] Run ['git', '-c', 'diff.ignoreSubmodules=none', '-c', 'core.quotepath=false', 'annex', 'find', '--anything', '--json', '--json-error-messages', '-c', 'annex.dotfiles=true', '--', 'subdir/1.tar'] (protocol_class=AnnexJsonProtocol) (cwd=/home/yoh/.tmp/datalad_temp_tree_rsua9kmg) 
+[DEBUG  ] Finished ['git', '-c', 'diff.ignoreSubmodules=none', '-c', 'core.quotepath=false', 'annex', 'find', '--anything', '--json', '--json-error-messages', '-c', 'annex.dotfiles=true', '--', 'subdir/1.tar'] with status 0 
+[DEBUG  ] Run ['git', '-c', 'diff.ignoreSubmodules=none', '-c', 'core.quotepath=false', 'annex', 'contentlocation', 'MD5E-s3584--bb87b72d411b7415410da27950d2a165.tar', '-c', 'annex.dotfiles=true'] (protocol_class=GeneratorStdOutErrCapture) (cwd=/home/yoh/.tmp/datalad_temp_tree_rsua9kmg) 
+[INFO   ] Adding content of the archive subdir/1.tar into annex AnnexRepo(/home/yoh/.tmp/datalad_temp_tree_rsua9kmg) 
+[DEBUG  ] Initiating clean cache for the archives under /home/yoh/.tmp/datalad_temp_tree_rsua9kmg/.git/datalad/tmp/archives 
+[DEBUG  ] Cache initialized 
+[DEBUG  ] Not initiating existing cache for the archives under /home/yoh/.tmp/datalad_temp_tree_rsua9kmg/.git/datalad/tmp/archives 
+[DEBUG  ] Cached directory for archive /home/yoh/.tmp/datalad_temp_tree_rsua9kmg/.git/annex/objects/gg/zf/MD5E-s3584--bb87b72d411b7415410da27950d2a165.tar/MD5E-s3584--bb87b72d411b7415410da27950d2a165.tar is fbab09b98e 
+[DEBUG  ] Run ['git', '-c', 'diff.ignoreSubmodules=none', '-c', 'core.quotepath=false', 'cat-file', 'blob', 'git-annex:remote.log'] (protocol_class=GeneratorStdOutErrCapture) (cwd=/home/yoh/.tmp/datalad_temp_tree_rsua9kmg) 
+[Level 11] CommandError: 'git -c diff.ignoreSubmodules=none -c core.quotepath=false cat-file blob git-annex:remote.log' failed with exitcode 128 [err: 'fatal: path 'remote.log' does not exist in 'git-annex''] 
+[DEBUG  ] Run ['git', '-c', 'diff.ignoreSubmodules=none', '-c', 'core.quotepath=false', 'cat-file', 'blob', 'git-annex:trust.log'] (protocol_class=GeneratorStdOutErrCapture) (cwd=/home/yoh/.tmp/datalad_temp_tree_rsua9kmg) 
+[Level 11] CommandError: 'git -c diff.ignoreSubmodules=none -c core.quotepath=false cat-file blob git-annex:trust.log' failed with exitcode 128 [err: 'fatal: path 'trust.log' does not exist in 'git-annex''] 
+[INFO   ] Initializing special remote datalad-archives 
+[DEBUG  ] Run ['git', '-c', 'diff.ignoreSubmodules=none', '-c', 'core.quotepath=false', 'annex', 'initremote', 'datalad-archives', 'encryption=none', 'type=external', 'autoenable=true', 'externaltype=datalad-archives', 'uuid=c04eb54b-4b4e-5755-8436-866b043170fa', '-c', 'annex.dotfiles=true'] (protocol_class=StdOutErrCapture) (cwd=/home/yoh/.tmp/datalad_temp_tree_rsua9kmg) 
+[DEBUG  ] Finished ['git', '-c', 'diff.ignoreSubmodules=none', '-c', 'core.quotepath=false', 'annex', 'initremote', 'datalad-archives', 'encryption=none', 'type=external', 'autoenable=true', 'externaltype=datalad-archives', 'uuid=c04eb54b-4b4e-5755-8436-866b043170fa', '-c', 'annex.dotfiles=true'] with status 0 
+[DEBUG  ] Run ['git', 'config', '-z', '-l', '--show-origin'] (protocol_class=StdOutErrCapture) (cwd=/home/yoh/.tmp/datalad_temp_tree_rsua9kmg) 
+[DEBUG  ] Finished ['git', 'config', '-z', '-l', '--show-origin'] with status 0 
+[DEBUG  ] Acquiring a lock /home/yoh/.tmp/datalad_temp_tree_rsua9kmg/.git/datalad/tmp/archives/fbab09b98e.extract-lck 
+[DEBUG  ] Acquired? lock /home/yoh/.tmp/datalad_temp_tree_rsua9kmg/.git/datalad/tmp/archives/fbab09b98e.extract-lck: True 
+[DEBUG  ] Extracting /home/yoh/.tmp/datalad_temp_tree_rsua9kmg/.git/annex/objects/gg/zf/MD5E-s3584--bb87b72d411b7415410da27950d2a165.tar/MD5E-s3584--bb87b72d411b7415410da27950d2a165.tar under /home/yoh/.tmp/datalad_temp_tree_rsua9kmg/.git/datalad/tmp/archives/fbab09b98e 
+[DEBUG  ] Run ['7z', 'x', '/home/yoh/.tmp/datalad_temp_tree_rsua9kmg/.git/annex/objects/gg/zf/MD5E-s3584--bb87b72d411b7415410da27950d2a165.tar/MD5E-s3584--bb87b72d411b7415410da27950d2a165.tar'] (protocol_class=KillOutput) (cwd=/home/yoh/.tmp/datalad_temp_tree_rsua9kmg/.git/datalad/tmp/archives/fbab09b98e) 
+[DEBUG  ] Finished ['7z', 'x', '/home/yoh/.tmp/datalad_temp_tree_rsua9kmg/.git/annex/objects/gg/zf/MD5E-s3584--bb87b72d411b7415410da27950d2a165.tar/MD5E-s3584--bb87b72d411b7415410da27950d2a165.tar'] with status 0 
+[DEBUG  ] Releasing lock /home/yoh/.tmp/datalad_temp_tree_rsua9kmg/.git/datalad/tmp/archives/fbab09b98e.extract-lck 
+[INFO   ] Start Extracting archive 
+[DEBUG  ] Adding /home/yoh/.tmp/datalad_temp_tree_rsua9kmg/subdir/.dataladiwgxvqzi/1/1.dat to annex pointing to dl+archive:MD5E-s3584--bb87b72d411b7415410da27950d2a165.tar#path=1/1.dat&size=5 and with options None 
+[DEBUG  ] Starting new runner for BatchedAnnex(command=['git', '-c', 'diff.ignoreSubmodules=none', '-c', 'core.quotepath=false', 'annex', 'addurl', '--with-files', '--json', '--json-error-messages', '--batch'], encoding=None, exception_on_timeout=False, last_request=None, output_proc=<function readline_json at 0x7f165f5adf80>, path=/home/yoh/.tmp/datalad_temp_tree_rsua9kmg, return_code=None, runner=None, stderr_output=b'', timeout=None, wait_timed_out=None) 
+[DEBUG  ] Run ['git', '-c', 'diff.ignoreSubmodules=none', '-c', 'core.quotepath=false', 'annex', 'addurl', '--with-files', '--json', '--json-error-messages', '--batch'] (protocol_class=BatchedCommandProtocol) (cwd=/home/yoh/.tmp/datalad_temp_tree_rsua9kmg) 
+[DEBUG  ] Starting new runner for BatchedAnnex(command=['git', '-c', 'diff.ignoreSubmodules=none', '-c', 'core.quotepath=false', 'annex', 'dropkey', '--force', '--json', '--json-error-messages', '--batch'], encoding=None, exception_on_timeout=False, last_request=None, output_proc=<function readline_json at 0x7f165f5adf80>, path=/home/yoh/.tmp/datalad_temp_tree_rsua9kmg, return_code=None, runner=None, stderr_output=b'', timeout=None, wait_timed_out=None) 
+[DEBUG  ] Run ['git', '-c', 'diff.ignoreSubmodules=none', '-c', 'core.quotepath=false', 'annex', 'dropkey', '--force', '--json', '--json-error-messages', '--batch'] (protocol_class=BatchedCommandProtocol) (cwd=/home/yoh/.tmp/datalad_temp_tree_rsua9kmg) 
+[DEBUG  ] Adding /home/yoh/.tmp/datalad_temp_tree_rsua9kmg/subdir/.dataladiwgxvqzi/1/file.txt to annex pointing to dl+archive:MD5E-s3584--bb87b72d411b7415410da27950d2a165.tar#path=1/file.txt&size=4 and with options None 
+[INFO   ] Finished adding subdir/1.tar: Files processed: 2, renamed: 2, removed: 2, +annex: 2 
+[DEBUG  ] Removing extracted and annexed files under /home/yoh/.tmp/datalad_temp_tree_rsua9kmg/subdir/.dataladiwgxvqzi 
+[DEBUG  ] Run ['git', '-c', 'diff.ignoreSubmodules=none', '-c', 'core.quotepath=false', 'rm', '--force', '-r', '--', 'subdir/.dataladiwgxvqzi'] (protocol_class=GeneratorStdOutErrCapture) (cwd=/home/yoh/.tmp/datalad_temp_tree_rsua9kmg) 
+[DEBUG  ] Query status of AnnexRepo('/home/yoh/.tmp/datalad_temp_tree_rsua9kmg') for all paths 
+[DEBUG  ] Run ['git', '-c', 'diff.ignoreSubmodules=none', '-c', 'core.quotepath=false', 'rev-parse', '--quiet', '--verify', 'HEAD^{commit}'] (protocol_class=GeneratorStdOutErrCapture) (cwd=/home/yoh/.tmp/datalad_temp_tree_rsua9kmg) 
+[DEBUG  ] AnnexRepo(/home/yoh/.tmp/datalad_temp_tree_rsua9kmg).get_content_info(...) 
+[DEBUG  ] Query repo: ['ls-files', '--stage', '-z'] 
+[DEBUG  ] Run ['git', '-c', 'diff.ignoreSubmodules=none', '-c', 'core.quotepath=false', 'ls-files', '--stage', '-z'] (protocol_class=GeneratorStdOutErrCapture) (cwd=/home/yoh/.tmp/datalad_temp_tree_rsua9kmg) 
+[DEBUG  ] Done query repo: ['ls-files', '--stage', '-z'] 
+[DEBUG  ] Done AnnexRepo(/home/yoh/.tmp/datalad_temp_tree_rsua9kmg).get_content_info(...) 
+[DEBUG  ] Run ['git', '-c', 'diff.ignoreSubmodules=none', '-c', 'core.quotepath=false', 'ls-files', '-z', '-m', '-d'] (protocol_class=GeneratorStdOutErrCapture) (cwd=/home/yoh/.tmp/datalad_temp_tree_rsua9kmg) 
+[DEBUG  ] AnnexRepo(/home/yoh/.tmp/datalad_temp_tree_rsua9kmg).get_content_info(...) 
+[DEBUG  ] Query repo: ['ls-tree', 'HEAD', '-z', '-r', '--full-tree', '-l'] 
+[DEBUG  ] Run ['git', '-c', 'diff.ignoreSubmodules=none', '-c', 'core.quotepath=false', 'ls-tree', 'HEAD', '-z', '-r', '--full-tree', '-l'] (protocol_class=GeneratorStdOutErrCapture) (cwd=/home/yoh/.tmp/datalad_temp_tree_rsua9kmg) 
+[DEBUG  ] Done query repo: ['ls-tree', 'HEAD', '-z', '-r', '--full-tree', '-l'] 
+[DEBUG  ] Done AnnexRepo(/home/yoh/.tmp/datalad_temp_tree_rsua9kmg).get_content_info(...) 
+[INFO   ]  Extracting archive 2 Files done in 0.872975 sec at 2.29102 Files/sec 
+[DEBUG  ] Cleaning up the cache for /home/yoh/.tmp/datalad_temp_tree_rsua9kmg/.git/annex/objects/gg/zf/MD5E-s3584--bb87b72d411b7415410da27950d2a165.tar/MD5E-s3584--bb87b72d411b7415410da27950d2a165.tar under /home/yoh/.tmp/datalad_temp_tree_rsua9kmg/.git/datalad/tmp/archives/fbab09b98e 
+[DEBUG  ] Cleaning up the stamp file for /home/yoh/.tmp/datalad_temp_tree_rsua9kmg/.git/annex/objects/gg/zf/MD5E-s3584--bb87b72d411b7415410da27950d2a165.tar/MD5E-s3584--bb87b72d411b7415410da27950d2a165.tar under /home/yoh/.tmp/datalad_temp_tree_rsua9kmg/.git/datalad/tmp/archives/fbab09b98e.stamp 
+add-archive-content(ok): /home/yoh/.tmp/datalad_temp_tree_rsua9kmg (dataset)
+
+```
+</details>
+
+[[!meta author=yoh]]
+[[!tag projects/repronim]]

Changed pages:

tips/clusters
Changed by: joeyh
Commit type: git
Date: 15:28:17 06/30/24

wording

diff --git a/doc/tips/clusters.mdwn b/doc/tips/clusters.mdwn
index 12a3a60d46..320e77f9a5 100644
--- a/doc/tips/clusters.mdwn
+++ b/doc/tips/clusters.mdwn
@@ -76,7 +76,7 @@ For example:
 
 By default, when a file is uploaded to a cluster, it is stored on every node of
 the cluster. To control which nodes to store to, the [[preferred_content]] of
-each node can be configured.
+each individual node can be configured.
 
 It's also a good idea to configure the preferred content of the cluster's
 gateway. To avoid files redundantly being stored on the gateway
@@ -119,8 +119,8 @@ in the git-annex branch. That tells other repositories about the cluster.
 	Started proxying for node3
 
 Operations that affect multiple nodes of a cluster can often be sped up by
-configuring annex.jobs in the repository that will serve the cluster to
-clients. In the example above, the nodes are all disk bound, so operating
+configuring annex.jobs in the gateway repository.
+In the example above, the nodes are all disk bound, so operating
 on more than one at a time will likely be faster.
 
     $ git config annex.jobs cpus

Changed pages:

tips/clusters
Changed by: joeyh
Commit type: git
Date: 15:14:13 06/30/24

list proxied remotes and cluster gateways in git-annex info
Wanted to also list a cluster's nodes when showing info for the cluster,
but that's hard because it needs getting the name of the proxying
remote, which is some prefix of the cluster's name, but if the names
contain dashes there's no good way to know which prefix it is.

diff --git a/Remote/Helper/Git.hs b/Remote/Helper/Git.hs
index c37d286df4..1567e7ae6a 100644
--- a/Remote/Helper/Git.hs
+++ b/Remote/Helper/Git.hs
@@ -1,6 +1,6 @@
 {- Utilities for git remotes.
  -
- - Copyright 2011-2014 Joey Hess <id@joeyh.name>
+ - Copyright 2011-2024 Joey Hess <id@joeyh.name>
  -
  - Licensed under the GNU AGPL version 3 or higher.
  -}
@@ -14,9 +14,13 @@ import Types.Availability
 import qualified Types.Remote as Remote
 import qualified Utility.RawFilePath as R
 import qualified Git.Config
+import Logs.Proxy
+import Types.Cluster
 
 import Data.Time.Clock.POSIX
 import System.PosixCompat.Files (modificationTime)
+import qualified Data.Map as M
+import qualified Data.Set as S
 
 repoCheap :: Git.Repo -> Bool
 repoCheap = not . Git.repoIsUrl
@@ -62,9 +66,26 @@ gitRepoInfo r = do
 		[] -> "never"
 		_ -> show $ posixSecondsToUTCTime $ realToFrac $ maximum mtimes
 	repo <- Remote.getRepo r
-	return
-		[ ("repository location", Git.repoLocation repo)
-		, ("proxied", Git.Config.boolConfig 
-			(isJust (remoteAnnexProxiedBy (Remote.gitconfig r))))
-		, ("last synced", lastsynctime)
+	let proxied = Git.Config.boolConfig $ isJust $
+		remoteAnnexProxiedBy (Remote.gitconfig r)
+	proxies <- getProxies
+	let proxying = S.toList $ fromMaybe mempty $
+		M.lookup (Remote.uuid r) proxies
+	let iscluster = isClusterUUID . proxyRemoteUUID
+	let proxyname p = Remote.name r ++ "-" ++ proxyRemoteName p
+	let proxynames = map proxyname $ filter (not . iscluster) proxying
+	let clusternames = map proxyname $ filter iscluster proxying
+	return $ catMaybes
+		[ Just ("repository location", Git.repoLocation repo)
+		, Just ("last synced", lastsynctime)
+		, Just ("proxied", proxied)
+		, if isClusterUUID (Remote.uuid r)
+			then Just ("cluster", Git.Config.boolConfig True)
+			else Nothing
+		, if null clusternames
+			then Nothing
+			else Just ("gateway to cluster", unwords clusternames)
+		, if null proxynames
+			then Nothing
+			else Just ("proxying", unwords proxynames)
 		]
diff --git a/doc/tips/clusters.mdwn b/doc/tips/clusters.mdwn
index a8ff6543ec..12a3a60d46 100644
--- a/doc/tips/clusters.mdwn
+++ b/doc/tips/clusters.mdwn
@@ -12,16 +12,21 @@ To use a cluster, your repository needs to have its gateway configured as a
 remote. Clusters can currently only be accessed via ssh. This gateway
 remote is added the same as any other git remote:
 
-    git remote add bigserver me@bigserver:annex
+    $ git remote add bigserver me@bigserver:annex
 
 The gateway publishes information about the cluster to the git-annex
 branch. So you may need to fetch from it to learn about the cluster:
 
-    git fetch bigserver
+    $ git fetch bigserver
 
 That will make available an additional remote for the cluster, eg
-"bigserver-mycluster", as well as some remotes for each node eg
-"bigserver-node1", "bigserver-node2", etc.
+"bigserver-mycluster", as well as some remotes for each node.
+
+	$ git annex info bigserver
+	...
+    gateway to cluster: bigserver-mycluster
+    proxying: bigserver-node1 bigserver-node2 bigserver-node3
+	...
 
 You can get files from the cluster without caring which node it comes
 from:

Changed pages:

todo/git-annex proxies
Changed by: joeyh
Commit type: git
Date: 21:14:18 06/28/24

todo

diff --git a/doc/todo/git-annex_proxies.mdwn b/doc/todo/git-annex_proxies.mdwn
index 8d803ec893..50eb5aeef3 100644
--- a/doc/todo/git-annex_proxies.mdwn
+++ b/doc/todo/git-annex_proxies.mdwn
@@ -45,6 +45,13 @@ For June's work on [[design/passthrough_proxy]], remaining todos:
   rather than PUT-FROM or ALREADY-HAVE. Verify that the client processes
   that ok and displays it to the user.
 
+* If a special remote outputs to stdout, or reads from stdin, that will
+  mess up the P2P protocol. Move the special remote proxying into a
+  separate process perhaps, which can be run with stdout and stdin
+  redirected? Or, fix any special remotes that might do that. Are
+  there any left? External special remotes certainly don't since that would
+  mess up their own protocol. Hook special remotes can though.
+
 * Streaming download from proxied special remotes. See design.
 
 * Check annex.diskreserve when proxying for special remotes.

Changed pages:

todo/git-annex proxies
Changed by: joeyh
Commit type: git
Date: 21:07:01 06/28/24

PUT to proxied special remote working
Still needs some work.
The reason that the waitv is necessary is because without it,
runNet loops back around and reads the next protocol message. But it's
not finished reading the whole bytestring yet, and so it reads some part
of it.

diff --git a/Annex/Proxy.hs b/Annex/Proxy.hs
index cbbbffadd5..60ab714f1d 100644
--- a/Annex/Proxy.hs
+++ b/Annex/Proxy.hs
@@ -15,8 +15,8 @@ import qualified Remote
 import qualified Types.Remote as Remote
 import qualified Remote.Git
 import Remote.Helper.Ssh (openP2PShellConnection', closeP2PShellConnection)
+import Annex.Content
 import Annex.Concurrent
-import Annex.Verify
 import Annex.Tmp
 import Utility.Tmp.Dir
 import Utility.Metered
@@ -51,14 +51,16 @@ proxySpecialRemoteSide clientmaxversion r = mkRemoteSide r $ do
 		liftIO (newTVarIO protoversion)
 	ihdl <- liftIO newEmptyTMVarIO
 	ohdl <- liftIO newEmptyTMVarIO
+	iwaitv <- liftIO newEmptyTMVarIO
+	owaitv <- liftIO newEmptyTMVarIO
 	endv <- liftIO newEmptyTMVarIO
 	worker <- liftIO . async =<< forkState
-		(proxySpecialRemote protoversion r ihdl ohdl endv)
+		(proxySpecialRemote protoversion r ihdl ohdl owaitv endv)
 	let remoteconn = P2PConnection
 		{ connRepo = Nothing
 		, connCheckAuth = const False
-		, connIhdl = P2PHandleTMVar ihdl
-		, connOhdl = P2PHandleTMVar ohdl
+		, connIhdl = P2PHandleTMVar ihdl iwaitv
+		, connOhdl = P2PHandleTMVar ohdl owaitv
 		, connIdent = ConnIdent (Just (Remote.name r))
 		}
 	let closeremoteconn = do
@@ -77,8 +79,9 @@ proxySpecialRemote
 	-> TMVar (Either L.ByteString Message)
 	-> TMVar (Either L.ByteString Message)
 	-> TMVar ()
+	-> TMVar ()
 	-> Annex ()
-proxySpecialRemote protoversion r ihdl ohdl endv = go
+proxySpecialRemote protoversion r ihdl ohdl owaitv endv = go
   where
 	go :: Annex ()
 	go = liftIO receivemessage >>= \case
@@ -97,7 +100,9 @@ proxySpecialRemote protoversion r ihdl ohdl endv = go
 				Right () -> liftIO $ sendmessage SUCCESS
 				Left err -> liftIO $ propagateerror err
 			go
-		Just (PUT af k) -> giveup "TODO PUT" -- XXX
+		Just (PUT (ProtoAssociatedFile af) k) -> do
+			proxyput af k
+			go
 		Just (GET offset (ProtoAssociatedFile af) k) -> do
 			proxyget offset af k
 			go
@@ -122,9 +127,10 @@ proxySpecialRemote protoversion r ihdl ohdl endv = go
 		Right (Right m) -> return (Just m)
 		Right (Left _b) -> giveup "unexpected ByteString received from P2P MVar"
 		Left () -> return Nothing
-	--receivebytestring = liftIO (atomically $ takeTMVar ohdl) >>= \case
-	--	Left b -> return b
-	--	Right _m -> giveup "did not receive ByteString from P2P MVar"
+	
+	receivebytestring = atomically (takeTMVar ohdl) >>= \case
+		Left b -> return b
+		Right _m -> giveup "did not receive ByteString from P2P MVar"
 
 	sendmessage m = atomically $ putTMVar ihdl (Right m)
 	
@@ -141,6 +147,59 @@ proxySpecialRemote protoversion r ihdl ohdl endv = go
 	withproxytmpfile k a = withOtherTmp $ \othertmpdir ->
 		withTmpDirIn (fromRawFilePath othertmpdir) "proxy" $ \tmpdir ->
 			a (toRawFilePath tmpdir P.</> keyFile k)
+			
+	proxyput af k = do
+		-- In order to send to the special remote, the key will
+		-- need to be inserted into the object directory.
+		-- It will be dropped again afterwards. Unless it's already
+		-- present there.
+		ifM (inAnnex k)
+			( tryNonAsync (Remote.storeKey r k af nullMeterUpdate) >>= \case
+				Right () -> liftIO $ sendmessage ALREADY_HAVE
+				Left err -> liftIO $ propagateerror err
+			, do
+				liftIO $ sendmessage $ PUT_FROM (Offset 0)
+				ifM receivedata
+					( do
+						tryNonAsync (Remote.storeKey r k af nullMeterUpdate) >>= \case
+							Right () -> do
+								depopulateobjectfile
+								liftIO $ sendmessage SUCCESS
+							Left err -> do
+								depopulateobjectfile
+								liftIO $ propagateerror err
+					, liftIO $ sendmessage FAILURE
+					)
+			)
+	  where
+		receivedata = withproxytmpfile k $ \tmpfile ->
+			liftIO receivemessage >>= \case
+				Just (DATA (Len _)) -> do
+					b <- liftIO receivebytestring
+					liftIO $ L.writeFile (fromRawFilePath tmpfile) b
+					-- Signal that the whole bytestring
+					-- has been stored.
+					liftIO $ atomically $ putTMVar owaitv ()
+					if protoversion > ProtocolVersion 1
+						then do
+							liftIO receivemessage >>= \case
+								Just (VALIDITY Valid) ->
+									populateobjectfile tmpfile
+								Just (VALIDITY Invalid) -> return False
+								_ -> giveup "protocol error"
+						else populateobjectfile tmpfile
+				_ -> giveup "protocol error"
+					
+		populateobjectfile tmpfile = 
+			getViaTmpFromDisk Remote.RetrievalAllKeysSecure Remote.DefaultVerify k af $ \dest -> do
+				unVerified $ do
+					liftIO $ renameFile
+						(fromRawFilePath tmpfile)
+						(fromRawFilePath dest)
+					return True
+						
+		depopulateobjectfile = void $ tryNonAsync $ 
+			lockContentForRemoval k noop removeAnnex
 
 	proxyget offset af k = withproxytmpfile k $ \tmpfile -> do
 		-- Don't verify the content from the remote,
@@ -148,7 +207,7 @@ proxySpecialRemote protoversion r ihdl ohdl endv = go
 		let vc = Remote.NoVerify
 		tryNonAsync (Remote.retrieveKeyFile r k af (fromRawFilePath tmpfile) nullMeterUpdate vc) >>= \case
 			Right v ->
-				ifM (verifyKeyContentPostRetrieval Remote.RetrievalAllKeysSecure vc v k tmpfile)
+				ifM (verifyKeyContentPostRetrieval Remote.RetrievalVerifiableKeysSecure vc v k tmpfile)
 					( liftIO $ senddata offset tmpfile
 					, liftIO $ sendmessage $
 						ERROR "verification of content failed"
diff --git a/P2P/IO.hs b/P2P/IO.hs
index 52a9b0a812..15e0dccde4 100644
--- a/P2P/IO.hs
+++ b/P2P/IO.hs
@@ -77,7 +77,7 @@ mkRunState mk = do
 
 data P2PHandle
 	= P2PHandle Handle
-	| P2PHandleTMVar (TMVar (Either L.ByteString Message))
+	| P2PHandleTMVar (TMVar (Either L.ByteString Message)) (TMVar ())
 
 data P2PConnection = P2PConnection
 	{ connRepo :: Maybe Repo
@@ -122,7 +122,7 @@ closeConnection conn = do
 	closehandle (connOhdl conn)
   where
 	closehandle (P2PHandle h) = hClose h
-	closehandle (P2PHandleTMVar _) = return ()
+	closehandle (P2PHandleTMVar _ _) = return ()
 
 -- Serves the protocol on a unix socket.
 --
@@ -190,7 +190,7 @@ runNet runst conn runner f = case f of
 				P2PHandle h -> tryNonAsync $ do
 					hPutStrLn h $ unwords (formatMessage m)
 					hFlush h
-				P2PHandleTMVar mv ->
+				P2PHandleTMVar mv _ ->
 					ifM (atomically (tryPutTMVar mv (Right m)))
 						( return $ Right ()
 						, return $ Left $ toException $
@@ -214,7 +214,7 @@ runNet runst conn runner f = case f of
 					Right (Just l) -> case parseMessage l of
 						Just m -> gotmessage m
 						Nothing -> runner (next Nothing)
-			P2PHandleTMVar mv -> 
+			P2PHandleTMVar mv _ -> 
 				liftIO (atomically (takeTMVar mv)) >>= \case
 					Right m -> gotmessage m
 					Left _b -> protoerr
@@ -230,8 +230,11 @@ runNet runst conn runner f = case f of
 					Right False -> return $ Left $
 						ProtoFailureMessage "short data write"
 					Left e -> return $ Left $ ProtoFailureException e
-			P2PHandleTMVar mv -> do
+			P2PHandleTMVar mv waitv -> do
 				liftIO $ atomically $ putTMVar mv (Left b)
+				-- Wait for the whole bytestring to be
+				-- processed. Necessary due to lazyiness.
+				liftIO $ atomically $ takeTMVar waitv
 				runner next
 	ReceiveBytes len p next ->
 		case connIhdl conn of
@@ -241,7 +244,7 @@ runNet runst conn runner f = case f of
 					Right b -> runner (next b)
 					Left e -> return $ Left $
 						ProtoFailureException e
-			P2PHandleTMVar mv ->
+			P2PHandleTMVar mv _ ->
 				liftIO (atomically (takeTMVar mv)) >>= \case
 					Left b -> runner (next b)

(Diff truncated)

Changed pages:

todo/git-annex proxies
Changed by: joeyh
Commit type: git
Date: 19:32:00 06/28/24

GET from proxied special remote
Working, but lots of room for improvement...
Without streaming, so there is a delay before download begins as the
file is retreived from the special remote.
And when resuming it retrieves the whole file from the special remote
*again*.
Also, if the special remote throws an exception, currently it
shows as "protocol error".

diff --git a/Annex/Proxy.hs b/Annex/Proxy.hs
index 21e950dd8b..cbbbffadd5 100644
--- a/Annex/Proxy.hs
+++ b/Annex/Proxy.hs
@@ -16,10 +16,15 @@ import qualified Types.Remote as Remote
 import qualified Remote.Git
 import Remote.Helper.Ssh (openP2PShellConnection', closeP2PShellConnection)
 import Annex.Concurrent
+import Annex.Verify
+import Annex.Tmp
+import Utility.Tmp.Dir
+import Utility.Metered
 
 import Control.Concurrent.STM
 import Control.Concurrent.Async
 import qualified Data.ByteString.Lazy as L
+import qualified System.FilePath.ByteString as P
 
 proxyRemoteSide :: ProtocolVersion -> Bypass -> Remote -> Annex RemoteSide
 proxyRemoteSide clientmaxversion bypass r
@@ -75,30 +80,34 @@ proxySpecialRemote
 	-> Annex ()
 proxySpecialRemote protoversion r ihdl ohdl endv = go
   where
-	go = receivemessage >>= \case
+	go :: Annex ()
+	go = liftIO receivemessage >>= \case
 		Just (CHECKPRESENT k) -> do
 			tryNonAsync (Remote.checkPresent r k) >>= \case
-				Right True -> sendmessage SUCCESS
-				Right False -> sendmessage FAILURE
-				Left err -> propagateerror err
+				Right True -> liftIO $ sendmessage SUCCESS
+				Right False -> liftIO $ sendmessage FAILURE
+				Left err -> liftIO $ propagateerror err
 			go
 		Just (LOCKCONTENT _) -> do
 			-- Special remotes do not support locking content.
-			sendmessage FAILURE
+			liftIO $ sendmessage FAILURE
 			go
 		Just (REMOVE k) -> do
 			tryNonAsync (Remote.removeKey r k) >>= \case
-				Right () -> sendmessage SUCCESS
-				Left err -> propagateerror err
+				Right () -> liftIO $ sendmessage SUCCESS
+				Left err -> liftIO $ propagateerror err
 			go
 		Just (PUT af k) -> giveup "TODO PUT" -- XXX
-		Just (GET offset af k) -> giveup "TODO GET" -- XXX
+		Just (GET offset (ProtoAssociatedFile af) k) -> do
+			proxyget offset af k
+			go
 		Just (BYPASS _) -> go
 		Just (CONNECT _) -> 
 			-- Not supported and the protocol ends here.
-			sendmessage $ CONNECTDONE (ExitFailure 1)	
+			liftIO $ sendmessage $ CONNECTDONE (ExitFailure 1)	
 		Just NOTIFYCHANGE -> do
-			sendmessage (ERROR "NOTIFYCHANGE unsupported for a special remote")
+			liftIO $ sendmessage $
+				ERROR "NOTIFYCHANGE unsupported for a special remote"
 			go
 		Just _ -> giveup "protocol error"
 		Nothing -> return ()
@@ -107,7 +116,7 @@ proxySpecialRemote protoversion r ihdl ohdl endv = go
 		liftIO $ atomically $ 
 			(Right <$> takeTMVar ohdl)
 				`orElse`
-			(Left <$> takeTMVar endv)
+			(Left <$> readTMVar endv)
 
 	receivemessage = getnextmessageorend >>= \case
 		Right (Right m) -> return (Just m)
@@ -117,8 +126,57 @@ proxySpecialRemote protoversion r ihdl ohdl endv = go
 	--	Left b -> return b
 	--	Right _m -> giveup "did not receive ByteString from P2P MVar"
 
-	sendmessage m = liftIO $ atomically $ putTMVar ihdl (Right m)
-	sendbytestring b = liftIO $ atomically $ putTMVar ihdl (Left b)
+	sendmessage m = atomically $ putTMVar ihdl (Right m)
+	
+	sendbytestring b = atomically $ putTMVar ihdl (Left b)
 
 	propagateerror err = sendmessage $ ERROR $
 		"proxied special remote reports: " ++ show err
+
+	-- Not using gitAnnexTmpObjectLocation because there might be
+	-- several concurrent GET and PUTs of the same key being proxied
+	-- from this special remote or others, and each needs to happen
+	-- independently. Also, this key is not getting added into the
+	-- local annex objects.
+	withproxytmpfile k a = withOtherTmp $ \othertmpdir ->
+		withTmpDirIn (fromRawFilePath othertmpdir) "proxy" $ \tmpdir ->
+			a (toRawFilePath tmpdir P.</> keyFile k)
+
+	proxyget offset af k = withproxytmpfile k $ \tmpfile -> do
+		-- Don't verify the content from the remote,
+		-- because the client will do its own verification.
+		let vc = Remote.NoVerify
+		tryNonAsync (Remote.retrieveKeyFile r k af (fromRawFilePath tmpfile) nullMeterUpdate vc) >>= \case
+			Right v ->
+				ifM (verifyKeyContentPostRetrieval Remote.RetrievalAllKeysSecure vc v k tmpfile)
+					( liftIO $ senddata offset tmpfile
+					, liftIO $ sendmessage $
+						ERROR "verification of content failed"
+					)
+			Left err -> liftIO $ propagateerror err
+	
+	senddata (Offset offset) f = do
+		size <- fromIntegral <$> getFileSize f
+		let n = max 0 (size - offset)
+		sendmessage $ DATA (Len n)
+		withBinaryFile (fromRawFilePath f) ReadMode $ \h -> do
+			hSeek h AbsoluteSeek offset
+			sendbs =<< L.hGetContents h
+			-- Important to keep the handle open until
+			-- the client responds. The bytestring
+			-- could still be lazily streaming out to
+			-- the client.
+			waitclientresponse
+	  where
+		sendbs bs = do
+			sendbytestring bs
+			when (protoversion > ProtocolVersion 0) $
+				sendmessage (VALIDITY Valid)
+			
+		waitclientresponse = 
+			receivemessage >>= \case
+				Just SUCCESS -> return ()
+				Just FAILURE -> return ()
+				Just _ -> giveup "protocol error"
+				Nothing -> return ()
+						
diff --git a/P2P/IO.hs b/P2P/IO.hs
index 643aafc4ec..52a9b0a812 100644
--- a/P2P/IO.hs
+++ b/P2P/IO.hs
@@ -241,10 +241,10 @@ runNet runst conn runner f = case f of
 					Right b -> runner (next b)
 					Left e -> return $ Left $
 						ProtoFailureException e
-			P2PHandleTMVar mv -> 
+			P2PHandleTMVar mv ->
 				liftIO (atomically (takeTMVar mv)) >>= \case
 					Left b -> runner (next b)
-					Right _m -> return $ Left $
+					Right _ -> return $ Left $
 						ProtoFailureMessage "protocol error"
 	CheckAuthToken _u t next -> do
 		let authed = connCheckAuth conn t
diff --git a/doc/todo/git-annex_proxies.mdwn b/doc/todo/git-annex_proxies.mdwn
index 73538cec42..de2960ad21 100644
--- a/doc/todo/git-annex_proxies.mdwn
+++ b/doc/todo/git-annex_proxies.mdwn
@@ -24,13 +24,18 @@ Planned schedule of work:
 
 For June's work on [[design/passthrough_proxy]], remaining todos:
 
-* Since proxying to special remotes is not supported yet, and won't be for
-  the first release, make it fail in a reasonable way.
+* Resuming an interrupted download from proxied special remote makes the proxy
+  re-download the whole content. It could instead keep some of the 
+  object files around when the client does not send SUCCESS. This would
+  use more disk, but without streaming, proxying a special remote already
+  needs some disk. And it could minimize to eg, the last 2 or so.
 
-- or -
+* If GET from a proxied special remote sends an ERROR with a message
+  from the special remote, currently the user sees "protocol error".
 
-* Proxying for special remotes.
-  Including encryption and chunking. See design for issues.
+* Implement PUT to proxied special remotes.
+
+* Streaming download from proxied special remotes. See design.
 
 # items deferred until later for [[design/passthrough_proxy]]
 
@@ -124,3 +129,6 @@ For June's work on [[design/passthrough_proxy]], remaining todos:
 
 * Proxied cluster nodes should have slightly higher cost than the cluster
   gateway. (done)
+
+* Basic support for proxying special remotes. (But not exporttree=yes ones
+  yet.) (done)

Changed pages:

privacy
Changed by: joeyh
Commit type: git
Date: 19:56:30 06/27/24

remove mention of XMPP which is no longer used

diff --git a/doc/privacy.mdwn b/doc/privacy.mdwn
index 5be735d1ad..4a14866a51 100644
--- a/doc/privacy.mdwn
+++ b/doc/privacy.mdwn
@@ -38,9 +38,6 @@ does not sanitize `--debug` output at all, so it may include the names of
 files or other repository details. You should review any debug or other
 output you post, and feel free to remove identifying information.
 
-Note that the git-annex assistant *does* sanitize XMPP protocol information
-logged when debugging is enabled.
-
 If you prefer not to post information publically, you can send a GPG
 encrypted mail to Joey Hess <id@joeyh.name> (gpg key ID 2512E3C7).
 Or you can post a public bug report, and send a followup email with private

Changed pages:

git-annex-initcluster

git-annex-updatecluster
Changed by: joeyh
Commit type: git
Date: 19:52:58 06/27/24

layout

diff --git a/doc/git-annex-extendcluster.mdwn b/doc/git-annex-extendcluster.mdwn
index fe7b8b0e3a..f6da205f73 100644
--- a/doc/git-annex-extendcluster.mdwn
+++ b/doc/git-annex-extendcluster.mdwn
@@ -34,8 +34,7 @@ after adding the new gateway repository as a remote.
 * [[git-annex-initcluster]](1)
 * [[git-annex-updatecluster]](1)
 * [[git-annex-updateproxy]](1)
-
-<https://git-annex.branchable.com/tips/clusters/>
+* <https://git-annex.branchable.com/tips/clusters/>
 
 # AUTHOR
 
diff --git a/doc/git-annex-initcluster.mdwn b/doc/git-annex-initcluster.mdwn
index 39b2b1c4c9..6895c2e866 100644
--- a/doc/git-annex-initcluster.mdwn
+++ b/doc/git-annex-initcluster.mdwn
@@ -29,8 +29,7 @@ documentation of that command for details about configuring nodes.
 * [[git-annex-extendcluster]](1)
 * [[git-annex-preferred-content]](1)
 * [[git-annex-updateproxy]](1)
-
-<https://git-annex.branchable.com/tips/clusters/>
+* <https://git-annex.branchable.com/tips/clusters/>
 
 # AUTHOR
 
diff --git a/doc/git-annex-updatecluster.mdwn b/doc/git-annex-updatecluster.mdwn
index d84d5593c8..13ab9ea24f 100644
--- a/doc/git-annex-updatecluster.mdwn
+++ b/doc/git-annex-updatecluster.mdwn
@@ -33,8 +33,7 @@ use [[git-annex-extendcluster]].
 * [[git-annex-initcluster]](1)
 * [[git-annex-extendcluster]](1)
 * [[git-annex-updateproxy]](1)
-
-<https://git-annex.branchable.com/tips/clusters/>
+* <https://git-annex.branchable.com/tips/clusters/>
 
 # AUTHOR

Changed pages:

git-annex-initcluster

git-annex-updatecluster

git-annex-updateproxy
Changed by: joeyh
Commit type: git
Date: 19:52:10 06/27/24

layout

diff --git a/doc/git-annex-extendcluster.mdwn b/doc/git-annex-extendcluster.mdwn
index 6367cf15af..fe7b8b0e3a 100644
--- a/doc/git-annex-extendcluster.mdwn
+++ b/doc/git-annex-extendcluster.mdwn
@@ -30,10 +30,10 @@ after adding the new gateway repository as a remote.
 
 # SEE ALSO
 
-[[git-annex]](1)
-[[git-annex-initcluster]](1)
-[[git-annex-updatecluster]](1)
-[[git-annex-updateproxy]](1)
+* [[git-annex]](1)
+* [[git-annex-initcluster]](1)
+* [[git-annex-updatecluster]](1)
+* [[git-annex-updateproxy]](1)
 
 <https://git-annex.branchable.com/tips/clusters/>
 
diff --git a/doc/git-annex-initcluster.mdwn b/doc/git-annex-initcluster.mdwn
index f96c0cd803..39b2b1c4c9 100644
--- a/doc/git-annex-initcluster.mdwn
+++ b/doc/git-annex-initcluster.mdwn
@@ -24,11 +24,11 @@ documentation of that command for details about configuring nodes.
 
 # SEE ALSO
 
-[[git-annex]](1)
-[[git-annex-updatecluster]](1)
-[[git-annex-extendcluster]](1)
-[[git-annex-preferred-content]](1)
-[[git-annex-updateproxy]](1)
+* [[git-annex]](1)
+* [[git-annex-updatecluster]](1)
+* [[git-annex-extendcluster]](1)
+* [[git-annex-preferred-content]](1)
+* [[git-annex-updateproxy]](1)
 
 <https://git-annex.branchable.com/tips/clusters/>
 
diff --git a/doc/git-annex-updatecluster.mdwn b/doc/git-annex-updatecluster.mdwn
index b72d6734b5..d84d5593c8 100644
--- a/doc/git-annex-updatecluster.mdwn
+++ b/doc/git-annex-updatecluster.mdwn
@@ -29,10 +29,10 @@ use [[git-annex-extendcluster]].
 
 # SEE ALSO
 
-[[git-annex]](1)
-[[git-annex-initcluster]](1)
-[[git-annex-extendcluster]](1)
-[[git-annex-updateproxy]](1)
+* [[git-annex]](1)
+* [[git-annex-initcluster]](1)
+* [[git-annex-extendcluster]](1)
+* [[git-annex-updateproxy]](1)
 
 <https://git-annex.branchable.com/tips/clusters/>
 
diff --git a/doc/git-annex-updateproxy.mdwn b/doc/git-annex-updateproxy.mdwn
index d6eb7c6398..10e26413f3 100644
--- a/doc/git-annex-updateproxy.mdwn
+++ b/doc/git-annex-updateproxy.mdwn
@@ -34,8 +34,8 @@ Proxies can only be accessed via ssh.
 
 # SEE ALSO
 
-[[git-annex]](1)
-[[git-annex-updatecluster]](1)
+* [[git-annex]](1)
+* [[git-annex-updatecluster]](1)
 
 # AUTHOR

Changed pages:

tips/clusters
Changed by: joeyh
Commit type: git
Date: 19:50:27 06/27/24

improve

diff --git a/doc/tips/clusters.mdwn b/doc/tips/clusters.mdwn
index 1a4a70ae97..a8ff6543ec 100644
--- a/doc/tips/clusters.mdwn
+++ b/doc/tips/clusters.mdwn
@@ -10,7 +10,7 @@ a node of the cluster.
 
 To use a cluster, your repository needs to have its gateway configured as a
 remote. Clusters can currently only be accessed via ssh. This gateway
-remote is added the same as any other remote:
+remote is added the same as any other git remote:
 
     git remote add bigserver me@bigserver:annex
 
@@ -102,7 +102,7 @@ In the example above, the three cluster nodes were configured like this:
 	$ git config remote.node2.annex-cluster-node mycluster
 	$ git config remote.node3.annex-cluster-node mycluster
 
-Finally, run `git-annex updatecluster` to record the cluster configuration
+Finally, run [[git-annex-updatecluster]] to record the cluster configuration
 in the git-annex branch. That tells other repositories about the cluster.
 	
 	$ git-annex updatecluster
@@ -148,7 +148,8 @@ accessed with ssh:
 
 Setting up the cluster in NYC is different, rather than using
 `git-annex initcluster` again (which would make a new, different
-cluster), we ask git-annex to extend the cluster from AMS:
+cluster), we ask git-annex to [[extend|git-annex-extendcluster]]
+the cluster from AMS:
 
     NYC$ git-annex extendcluster AMS mycluster

Changed pages:

todo/git-annex proxies
Changed by: joeyh
Commit type: git
Date: 19:44:11 06/27/24

merged the proxy branch into master!

diff --git a/doc/todo/git-annex_proxies.mdwn b/doc/todo/git-annex_proxies.mdwn
index de0c11d161..73538cec42 100644
--- a/doc/todo/git-annex_proxies.mdwn
+++ b/doc/todo/git-annex_proxies.mdwn
@@ -22,8 +22,6 @@ Planned schedule of work:
 
 # work notes
 
-In development on the `proxy` branch.
-
 For June's work on [[design/passthrough_proxy]], remaining todos:
 
 * Since proxying to special remotes is not supported yet, and won't be for

Changed pages:

git-annex-initcluster

git-annex-updatecluster

links/key concepts

tips/clusters
Changed by: joeyh
Commit type: git
Date: 19:41:38 06/27/24

move clusters page to tips
also add a section on the front page highlighting major new features

diff --git a/doc/git-annex-extendcluster.mdwn b/doc/git-annex-extendcluster.mdwn
index 9674cc9d15..6367cf15af 100644
--- a/doc/git-annex-extendcluster.mdwn
+++ b/doc/git-annex-extendcluster.mdwn
@@ -35,7 +35,7 @@ after adding the new gateway repository as a remote.
 [[git-annex-updatecluster]](1)
 [[git-annex-updateproxy]](1)
 
-<https://git-annex.branchable.com/clusters/>
+<https://git-annex.branchable.com/tips/clusters/>
 
 # AUTHOR
 
diff --git a/doc/git-annex-initcluster.mdwn b/doc/git-annex-initcluster.mdwn
index 68e0e904ca..f96c0cd803 100644
--- a/doc/git-annex-initcluster.mdwn
+++ b/doc/git-annex-initcluster.mdwn
@@ -30,7 +30,7 @@ documentation of that command for details about configuring nodes.
 [[git-annex-preferred-content]](1)
 [[git-annex-updateproxy]](1)
 
-<https://git-annex.branchable.com/clusters/>
+<https://git-annex.branchable.com/tips/clusters/>
 
 # AUTHOR
 
diff --git a/doc/git-annex-updatecluster.mdwn b/doc/git-annex-updatecluster.mdwn
index b40b417f73..b72d6734b5 100644
--- a/doc/git-annex-updatecluster.mdwn
+++ b/doc/git-annex-updatecluster.mdwn
@@ -34,7 +34,7 @@ use [[git-annex-extendcluster]].
 [[git-annex-extendcluster]](1)
 [[git-annex-updateproxy]](1)
 
-<https://git-annex.branchable.com/clusters/>
+<https://git-annex.branchable.com/tips/clusters/>
 
 # AUTHOR
 
diff --git a/doc/links/key_concepts.mdwn b/doc/links/key_concepts.mdwn
index 0c2e1ddf74..b08384d400 100644
--- a/doc/links/key_concepts.mdwn
+++ b/doc/links/key_concepts.mdwn
@@ -4,6 +4,10 @@
 * [[how_it_works]]
 * [[special_remotes]]
 * [[workflows|workflow]]
-* [[sync]]
 * [[preferred_content]]
-* [[clusters]]
+* [[sync]]
+
+### new features
+
+* [[tips/clusters]]
+* [[git-remote-annex|tips/storing_a_git_repository_on_any_special_remote]]
diff --git a/doc/clusters.mdwn b/doc/tips/clusters.mdwn
similarity index 100%
rename from doc/clusters.mdwn
rename to doc/tips/clusters.mdwn

Changed pages:

git-annex-extendcluster
Changed by: joeyh
Commit type: git
Date: 19:34:45 06/27/24

make extendcluster also updatecluster
This avoids the user forgetting to do it and simplifies the
documentation.

diff --git a/Command/ExtendCluster.hs b/Command/ExtendCluster.hs
index 6fa248d57a..21e65903f1 100644
--- a/Command/ExtendCluster.hs
+++ b/Command/ExtendCluster.hs
@@ -15,6 +15,7 @@ import Types.Cluster
 import Config
 import Types.GitConfig
 import qualified Remote
+import qualified Command.UpdateCluster
 
 import qualified Data.Map as M
 
@@ -27,7 +28,9 @@ seek (remotename:clustername:[]) = Remote.byName (Just clusterremotename) >>= \c
 	Just clusterremote -> Remote.byName (Just remotename) >>= \case
 		Just gatewayremote -> 
 			case mkClusterUUID (Remote.uuid clusterremote) of
-				Just cu -> commandAction $ start cu clustername gatewayremote
+				Just cu -> do
+					commandAction $ start cu clustername gatewayremote
+					Command.UpdateCluster.seek []
 				Nothing -> giveup $ clusterremotename 
 					++ " is not a cluster remote."
 		Nothing -> giveup $ "No remote named " ++ remotename ++ " exists."
diff --git a/doc/clusters.mdwn b/doc/clusters.mdwn
index e26163a88e..1a4a70ae97 100644
--- a/doc/clusters.mdwn
+++ b/doc/clusters.mdwn
@@ -168,7 +168,6 @@ for NYC, and extending the cluster to there as well:
     AMS$ git remote add NYC me@nyc.example.com:annex
     AMS$ git-annex sync NYC
     NYC$ git-annex extendcluster NYC mycluster
-    AMS$ git-annex updatecluster
 
 A user can now add either AMS or NYC as a remote, and will have access
 to the entire cluster as either `AMS-mycluster` or `NYC-mycluster`.
diff --git a/doc/git-annex-extendcluster.mdwn b/doc/git-annex-extendcluster.mdwn
index 79796fe6e8..9674cc9d15 100644
--- a/doc/git-annex-extendcluster.mdwn
+++ b/doc/git-annex-extendcluster.mdwn
@@ -17,8 +17,12 @@ The `clustername` parameter is the name of the cluster.
 
 The next step after running this command is to configure
 any additional cluster nodes that this gateway serves to the cluster,
-then run [[git-annex-updatecluster]] on each gateway. 
-See the documentation of that command for details about configuring nodes.
+then run [[git-annex-updatecluster]]. See the documentation of that
+command for details about configuring nodes.
+
+After running this command in the new gateway repository, it typically
+also needs to be run in the other gateway repositories as well, 
+after adding the new gateway repository as a remote.
 
 # OPTIONS

Changed pages:

todo/git-annex proxies
Changed by: joeyh
Commit type: git
Date: 19:28:10 06/27/24

update

diff --git a/doc/todo/git-annex_proxies.mdwn b/doc/todo/git-annex_proxies.mdwn
index 29f662e6bf..de0c11d161 100644
--- a/doc/todo/git-annex_proxies.mdwn
+++ b/doc/todo/git-annex_proxies.mdwn
@@ -11,7 +11,7 @@ repositories.
 Joey has received funding to work on this.
 Planned schedule of work:
 
-* June: git-annex proxy
+* June: git-annex proxies and clusters
 * July, part 1: git-annex proxy support for exporttree
 * July, part 2: p2p protocol over http
 * August: balanced preferred content
@@ -29,9 +29,12 @@ For June's work on [[design/passthrough_proxy]], remaining todos:
 * Since proxying to special remotes is not supported yet, and won't be for
   the first release, make it fail in a reasonable way.
 
+- or -
+
 * Proxying for special remotes.
+  Including encryption and chunking. See design for issues.
 
-* Encryption and chunking. See design for issues.
+# items deferred until later for [[design/passthrough_proxy]]
 
 * Indirect uploads when proxying for special remote
   (to be considered). See design.
@@ -79,7 +82,7 @@ For June's work on [[design/passthrough_proxy]], remaining todos:
 * Proxy should update location tracking information for proxied remotes,
   so it is available to other users who sync with it. (done)
 
-* Implement `git-annex updatecluster` command (done)
+* Implement `git-annex initcluster` and `git-annex updatecluster` commands (done)
 
 * Implement cluster UUID insertation on location log load, and removal
   on location log store. (done)

Changed pages:

todo/git-annex proxies
Changed by: joeyh
Commit type: git
Date: 19:21:03 06/27/24

give proxied cluster nodes a higher cost than the cluster gateway
This makes eg git-annex get default to using the cluster rather than an
arbitrary node, which is better UI.
The actual cost of accessing a proxied node vs using the cluster is
basically the same. But using the cluster allows smarter load-balancing
to be done on the cluster.

diff --git a/Remote/Git.hs b/Remote/Git.hs
index 89e0da38c1..844beb3b2e 100644
--- a/Remote/Git.hs
+++ b/Remote/Git.hs
@@ -176,6 +176,7 @@ configRead autoinit r = do
 			Just r' -> return r'
 		_ -> return r
 
+
 gen :: Git.Repo -> UUID -> RemoteConfig -> RemoteGitConfig -> RemoteStateHandle -> Annex (Maybe Remote)
 gen r u rc gc rs
 	-- Remote.GitLFS may be used with a repo that is also encrypted
@@ -186,10 +187,9 @@ gen r u rc gc rs
 		Nothing -> do
 			st <- mkState r u gc
 			c <- parsedRemoteConfig remote rc
-			go st c <$> remoteCost gc c defcst
+			go st c <$> remoteCost gc c (defaultRepoCost r)
 		Just addr -> Remote.P2P.chainGen addr r u rc gc rs
   where
-	defcst = if repoCheap r then cheapRemoteCost else expensiveRemoteCost
 	go st c cst = Just new
 	  where
 		new = Remote 
@@ -229,6 +229,11 @@ gen r u rc gc rs
 			, remoteStateHandle = rs
 			}
 
+defaultRepoCost :: Git.Repo -> Cost
+defaultRepoCost r
+	| repoCheap r = cheapRemoteCost
+	| otherwise = expensiveRemoteCost
+
 unavailable :: Git.Repo -> UUID -> RemoteConfig -> RemoteGitConfig -> RemoteStateHandle -> Annex (Maybe Remote)
 unavailable r = gen r'
   where
@@ -854,12 +859,17 @@ listProxied proxies rs = concat <$> mapM go rs
 		-- that cluster does not need to be synced with
 		-- by default, because syncing with the cluster will
 		-- effectively sync with all of its nodes.
+		--
+		-- Also, give it a slightly higher cost than the
+		-- cluster by default, to encourage using the cluster.
 		adjustclusternode clusters =
 			case M.lookup (ClusterNodeUUID (proxyRemoteUUID p)) (clusterNodeUUIDs clusters) of
 				Just cs
 					| any (\c -> S.member (fromClusterUUID c) proxieduuids) (S.toList cs) ->
 						addremoteannexfield SyncField
 							[Git.ConfigValue $ Git.Config.boolConfig' False]
+						. addremoteannexfield CostField 
+							[Git.ConfigValue $ encodeBS $ show $ defaultRepoCost r + 0.1]
 				_ -> id
 
 		proxieduuids = S.map proxyRemoteUUID proxied
diff --git a/doc/todo/git-annex_proxies.mdwn b/doc/todo/git-annex_proxies.mdwn
index a53197f8c1..29f662e6bf 100644
--- a/doc/todo/git-annex_proxies.mdwn
+++ b/doc/todo/git-annex_proxies.mdwn
@@ -41,14 +41,17 @@ For June's work on [[design/passthrough_proxy]], remaining todos:
   eg prefer to avoid using remotes that are doing other transfers at the
   same time.
 
-* The cost of a cluster and of its proxied nodes is currently all the same.
-  It would make sense for proxied nodes that are accessed via an intermedia
-  gateway to have a higher cost than proxied nodes that are accessed via
-  the remote gateway. And proxied nodes should generally have a higher cost
-  than the cluster, so that git-annex defaults to using the cluster.
-  (The cost of accessing a proxied node vs using the cluster is the same,
-  but using the cluster allows smarter load-balancing to be done on the
-  cluster. It also makes the UI not mention individual nodes.)
+* The cost of a proxied node that is accessed via an intermediate gateway
+  is currently the same as a node accessed via the cluster gateway.
+  To fix this, there needs to be some way to tell how many hops through
+  gateways it takes to get to a node. Currently the only way is to
+  guess based on number of dashes in the node name, which is not satisfying.
+
+  Even counting hops is not very satisfying, one cluster gateway could
+  be much more expensive to traverse than another one.
+
+  If seriously tackling this, it might be worth making enough information
+  available to use spanning tree protocol for routing inside clusters.
 
 * Optimise proxy speed. See design for ideas.
 
@@ -117,3 +120,6 @@ For June's work on [[design/passthrough_proxy]], remaining todos:
   protocol messages on to any remotes that have the same UUID as
   the cluster. Needs extension to P2P protocol to avoid cycles.
   (done)
+
+* Proxied cluster nodes should have slightly higher cost than the cluster
+  gateway. (done)

Changed pages:

todo/git-annex proxies
Changed by: joeyh
Commit type: git
Date: 18:36:55 06/27/24

GET and CHECKPRESENT amoung lowest cost cluster nodes
Before it was using a node that might have had a higher cost.
Also threw in a random selection from amoung the low cost nodes. Of
course this is a poor excuse for load balancing, but it's better than
nothing. Most of the time...

diff --git a/Annex/Cluster.hs b/Annex/Cluster.hs
index 1bcfa04f14..2c397beeb7 100644
--- a/Annex/Cluster.hs
+++ b/Annex/Cluster.hs
@@ -27,6 +27,7 @@ import qualified Types.Remote as Remote
 
 import qualified Data.Map as M
 import qualified Data.Set as S
+import System.Random
 
 {- Proxy to a cluster. -}
 proxyCluster 
@@ -75,7 +76,7 @@ clusterProxySelector clusteruuid protocolversion (Bypass bypass) = do
 	nodeuuids <- (fromMaybe S.empty . M.lookup clusteruuid . clusterUUIDs)
 		<$> getClusters
 	myclusters <- annexClusters <$> Annex.getGitConfig
-	allremotes <- remoteList
+	allremotes <- concat . Remote.byCost <$> remoteList
 	hereu <- getUUID
 	let bypass' = S.insert hereu bypass
 	let clusterremotes = filter (isnode bypass' allremotes nodeuuids myclusters) allremotes
@@ -94,8 +95,8 @@ clusterProxySelector clusteruuid protocolversion (Bypass bypass) = do
 		-- skipping nodes where it's not preferred content.
 		, proxyPUT = \af k -> do
 			locs <- S.fromList <$> loggedLocations k
-			let l = filter (flip S.notMember locs . remoteUUID) nodes
-			l' <- filterM (\n -> isPreferredContent (Just (remoteUUID n)) mempty (Just k) af True) l
+			let l = filter (flip S.notMember locs . Remote.uuid . remote) nodes
+			l' <- filterM (\n -> isPreferredContent (Just (Remote.uuid (remote n))) mempty (Just k) af True) l
 			-- PUT to no nodes doesn't work, so fall
 			-- back to all nodes.
 			return $ nonempty [l', l] nodes
@@ -146,11 +147,19 @@ clusterProxySelector clusteruuid protocolversion (Bypass bypass) = do
 	
 	nodecontaining nodes k = do
 		locs <- S.fromList <$> loggedLocations k
-		case filter (flip S.member locs . remoteUUID) nodes of
-			-- For now, pick the first node that has the
-			-- content. Load balancing would be nice..
-			(r:_) -> return (Just r)
+		case filter (flip S.member locs . Remote.uuid . remote) nodes of
 			[] -> return Nothing
+			(node:[]) -> return (Just node)
+			(node:rest) ->
+				-- The list of nodes is ordered by cost.
+				-- Use any of the ones with equally low
+				-- cost.
+				let lowestcost = Remote.cost (remote node)
+				    samecost = node : takeWhile (\n -> Remote.cost (remote n) == lowestcost) rest
+				in do
+					n <- getStdRandom $
+						randomR (0, length samecost - 1)
+					return (Just (samecost !! n))
 		
 	nonempty (l:ls) fallback
 		| null l = nonempty ls fallback
diff --git a/Annex/Proxy.hs b/Annex/Proxy.hs
index 747f393d6d..759c526162 100644
--- a/Annex/Proxy.hs
+++ b/Annex/Proxy.hs
@@ -11,18 +11,16 @@ import Annex.Common
 import P2P.Proxy
 import P2P.Protocol
 import P2P.IO
-import qualified Remote
 import Remote.Helper.Ssh (openP2PShellConnection', closeP2PShellConnection)
 
 -- FIXME: Support special remotes.
 proxySshRemoteSide :: ProtocolVersion -> Bypass -> Remote -> Annex RemoteSide
-proxySshRemoteSide clientmaxversion bypass remote = 
-	mkRemoteSide (Remote.uuid remote) $
-		openP2PShellConnection' remote clientmaxversion bypass >>= \case
-			Just conn@(OpenConnection (remoterunst, remoteconn, _)) ->
-				return $ Just 
-					( remoterunst
-					, remoteconn
-					, void $ liftIO $ closeP2PShellConnection conn
-					)
-			_  -> return Nothing
+proxySshRemoteSide clientmaxversion bypass r = mkRemoteSide r $
+	openP2PShellConnection' r clientmaxversion bypass >>= \case
+		Just conn@(OpenConnection (remoterunst, remoteconn, _)) ->
+			return $ Just 
+				( remoterunst
+				, remoteconn
+				, void $ liftIO $ closeP2PShellConnection conn
+				)
+		_  -> return Nothing
diff --git a/Command/P2PStdIO.hs b/Command/P2PStdIO.hs
index f72a1d314f..dc3e081d81 100644
--- a/Command/P2PStdIO.hs
+++ b/Command/P2PStdIO.hs
@@ -60,14 +60,14 @@ performLocal theiruuid servermode = do
 	p2pErrHandler (const p2pDone) (runFullProto runst conn server)
 
 performProxy :: UUID -> P2P.ServerMode -> Remote -> CommandPerform
-performProxy clientuuid servermode remote = do
+performProxy clientuuid servermode r = do
 	clientside <- proxyClientSide clientuuid
-	getClientProtocolVersion (Remote.uuid remote) clientside 
+	getClientProtocolVersion (Remote.uuid r) clientside 
 		(withclientversion clientside)
 		p2pErrHandler
   where
 	withclientversion clientside (Just (clientmaxversion, othermsg)) = do
-		remoteside <- proxySshRemoteSide clientmaxversion mempty remote
+		remoteside <- proxySshRemoteSide clientmaxversion mempty r
 		protocolversion <- either (const (min P2P.maxProtocolVersion clientmaxversion)) id
 			<$> runRemoteSide remoteside 
 				(P2P.net P2P.getProtocolVersion)
@@ -77,7 +77,7 @@ performProxy clientuuid servermode remote = do
 		concurrencyconfig <- noConcurrencyConfig
 		let runproxy othermsg' = proxy closer proxymethods
 			servermode clientside
-			(Remote.uuid remote)
+			(Remote.uuid r)
 			(singleProxySelector remoteside)
 			concurrencyconfig
 			protocolversion othermsg' p2pErrHandler
diff --git a/P2P/Proxy.hs b/P2P/Proxy.hs
index 84da1d6336..89364133e3 100644
--- a/P2P/Proxy.hs
+++ b/P2P/Proxy.hs
@@ -18,6 +18,7 @@ import Utility.Metered
 import Git.FilePath
 import Types.Concurrency
 import Annex.Concurrent
+import qualified Remote
 
 import Data.Either
 import Control.Concurrent.STM
@@ -32,14 +33,14 @@ type ProtoCloser = Annex ()
 data ClientSide = ClientSide RunState P2PConnection
 
 data RemoteSide = RemoteSide
-	{ remoteUUID :: UUID
+	{ remote :: Remote
 	, remoteConnect :: Annex (Maybe (RunState, P2PConnection, ProtoCloser))
 	, remoteTMVar :: TMVar (RunState, P2PConnection, ProtoCloser)
 	}
 
-mkRemoteSide :: UUID -> Annex (Maybe (RunState, P2PConnection, ProtoCloser)) -> Annex RemoteSide
-mkRemoteSide remoteuuid remoteconnect = RemoteSide
-	<$> pure remoteuuid
+mkRemoteSide :: Remote -> Annex (Maybe (RunState, P2PConnection, ProtoCloser)) -> Annex RemoteSide
+mkRemoteSide r remoteconnect = RemoteSide
+	<$> pure r
 	<*> pure remoteconnect
 	<*> liftIO (atomically newEmptyTMVar)
 
@@ -328,9 +329,9 @@ proxy proxydone proxymethods servermode (ClientSide clientrunst clientconn) remo
 				net $ sendMessage message
 				net receiveMessage >>= return . \case
 					Just SUCCESS ->
-						Just (True, [remoteUUID r])
+						Just (True, [Remote.uuid (remote r)])
 					Just (SUCCESS_PLUS us) -> 
-						Just (True, remoteUUID r:us)
+						Just (True, Remote.uuid (remote r):us)
 					Just FAILURE ->
 						Just (False, [])
 					Just (FAILURE_PLUS us) ->
@@ -355,7 +356,7 @@ proxy proxydone proxymethods servermode (ClientSide clientrunst clientconn) remo
 		withDATA (relayGET remoteside)
 
 	handlePUT (remoteside:[]) k message
-		| remoteUUID remoteside == remoteuuid =
+		| Remote.uuid (remote remoteside) == remoteuuid =
 			getresponse (runRemoteSide remoteside) message $ \resp -> case resp of
 				ALREADY_HAVE -> protoerrhandler proxynextclientmessage $
 					client $ net $ sendMessage resp
@@ -390,10 +391,10 @@ proxy proxydone proxymethods servermode (ClientSide clientrunst clientconn) remo
 			proxynextclientmessage ()
 
 	relayPUTRecord k remoteside SUCCESS = do
-		addedContent proxymethods (remoteUUID remoteside) k
-		return $ Just [remoteUUID remoteside]
+		addedContent proxymethods (Remote.uuid (remote remoteside)) k
+		return $ Just [Remote.uuid (remote remoteside)]
 	relayPUTRecord k remoteside (SUCCESS_PLUS us) = do
-		let us' = remoteUUID remoteside : us
+		let us' = (Remote.uuid (remote remoteside)) : us
 		forM_ us' $ \u ->
 			addedContent proxymethods u k
 		return $ Just us'
@@ -425,7 +426,7 @@ proxy proxydone proxymethods servermode (ClientSide clientrunst clientconn) remo
 				else protoerrhandler proxynextclientmessage $
 					client $ net $ sendMessage $ ALREADY_HAVE_PLUS $
 						filter (/= remoteuuid) $
-							map remoteUUID (lefts (rights l))
+							map (Remote.uuid . remote) (lefts (rights l))
 			else if null (rights l)
 				-- no response from any remote
 				then proxydone
@@ -439,11 +440,11 @@ proxy proxydone proxymethods servermode (ClientSide clientrunst clientconn) remo
 		let totallen = datalen + minoffset
 		-- Tell each remote how much data to expect, depending
 		-- on the remote's offset.
-		rs <- forMC concurrencyconfig remotes $ \remote@(remoteside, remoteoffset) ->
+		rs <- forMC concurrencyconfig remotes $ \r@(remoteside, remoteoffset) ->

(Diff truncated)

Changed pages:

design/passthrough proxy

todo/git-annex proxies
Changed by: joeyh
Commit type: git
Date: 17:40:09 06/27/24

update

diff --git a/doc/design/passthrough_proxy.mdwn b/doc/design/passthrough_proxy.mdwn
index 8d62649c13..4db69b847a 100644
--- a/doc/design/passthrough_proxy.mdwn
+++ b/doc/design/passthrough_proxy.mdwn
@@ -318,22 +318,24 @@ This does mean that cycles need to be prevented. See section below.
 
 ## speed
 
-A passthrough proxy should be as fast as possible so as not to add overhead
+A proxy should be as fast as possible so as not to add overhead
 to a file retrieve, store, or checkpresent. This probably means that
-it keeps TCP connections open to each host in the cluster. It might use a
+it keeps TCP connections open to each host. It might use a
 protocol with less overhead than ssh.
 
-In the case of checkpresent, it would be possible for the proxy to not
-communicate with the cluster to check that the data is still present on it.
-As long as all access is intermediated via the proxy, its git-annex branch
-could be relied on to always be correct, in theory. Proving that theory,
-making sure to account for all possible race conditions and other scenarios,
-would be necessary for such an optimisation.
-
-Another way the proxy could speed things up is to cache some subset of
-content. Eg, analize what files are typically requested, and store another
-copy of those on the proxy. Perhaps prioritize storing smaller files, where
-latency tends to swamp transfer speed.
+In the case of checkpresent, it would be possible for the gateway to not
+communicate with cluster nodes to check that the data is still present
+in the cluster. As long as all access is intermediated via a single gateway, 
+its git-annex branch could be relied on to always be correct, in theory.
+Proving that theory, making sure to account for all possible race conditions
+and other scenarios, would be necessary for such an optimisation. This
+would not work for multi-gateway clusters unless the gateways were kept in
+sync about locations, which they currently are not.
+
+Another way the cluster gateway could speed things up is to cache some
+subset of content. Eg, analize what files are typically requested, and
+store another copy of those on the proxy. Perhaps prioritize storing
+smaller files, where latency tends to swamp transfer speed.
 
 ## proxying to special remotes
 
diff --git a/doc/todo/git-annex_proxies.mdwn b/doc/todo/git-annex_proxies.mdwn
index d430de9433..e803f40600 100644
--- a/doc/todo/git-annex_proxies.mdwn
+++ b/doc/todo/git-annex_proxies.mdwn
@@ -33,6 +33,9 @@ For June's work on [[design/passthrough_proxy]], remaining todos:
 
 * Encryption and chunking. See design for issues.
 
+* Indirect uploads when proxying for special remote
+  (to be considered). See design.
+
 * Getting a key from a cluster currently always selects the lowest cost
   remote, and always the same remote if cost is the same. Should
   round-robin amoung remotes, and prefer to avoid using remotes that
@@ -45,8 +48,6 @@ For June's work on [[design/passthrough_proxy]], remaining todos:
   Library to use:
   <https://hackage.haskell.org/package/hsyscall-0.4/docs/System-Syscall.html>
 
-* Indirect uploads (to be considered). See design.
-
 * Support using a proxy when its url is a P2P address.
   (Eg tor-annex remotes.)

Changed pages:

todo/git-annex proxies
Changed by: joeyh
Commit type: git
Date: 17:34:24 06/27/24

remove viconfig item
it works when run on a client that has the cluster gateway as a remote,
just not when on the cluster gateway

diff --git a/doc/todo/git-annex_proxies.mdwn b/doc/todo/git-annex_proxies.mdwn
index 0dc0836bc2..d430de9433 100644
--- a/doc/todo/git-annex_proxies.mdwn
+++ b/doc/todo/git-annex_proxies.mdwn
@@ -50,9 +50,6 @@ For June's work on [[design/passthrough_proxy]], remaining todos:
 * Support using a proxy when its url is a P2P address.
   (Eg tor-annex remotes.)
 
-* `viconfig` support for setting preferred content, group, 
-  and description of clusters
-
 # completed items for June's work on [[design/passthrough_proxy]]:
 
 * UUID discovery via git-annex branch. Add a log file listing UUIDs

Changed pages:

todo/git-annex proxies
Changed by: joeyh
Commit type: git
Date: 17:33:04 06/27/24

document various multi-gateway cluster considerations
Perhaps this will avoid me needing to eg, implement spanning tree
protocol. ;-)

diff --git a/doc/clusters.mdwn b/doc/clusters.mdwn
index 4f1062ab7f..e26163a88e 100644
--- a/doc/clusters.mdwn
+++ b/doc/clusters.mdwn
@@ -192,13 +192,27 @@ Notice that remotes for cluster nodes have names indicating the path through
 the cluster used to access them. For example, "AMS-NYC-node3" is accessed via
 the AMS gateway, which then relays to NYC where node3 is located.
 
-## cluster topologies
+## considerations for multi-gateway clusters
+
+When a cluster has multiple gateways, nothing keeps the git repositories on
+the gateways in sync. A branch pushed to one gateway will not be able to
+be pulled from another one. And gateways only learn about the locations of
+keys that are uploaded to the cluster via them. So in the example above,
+after an upload to AMS-mycluster, NYC-mycluster will only know that the
+key is stored in its nodes, but won't know that it's stored in nodes
+behind AMS. So, it's best to have a single git repository that is synced
+with, or perhaps run [[git-annex-remotedaemon]] on each gateway to keep
+its git repository in sync with the other gateways.
 
 Clusters can be constructed with any number of gateways, and any internal
-topology of connections between gateways. 
-
-There must always be a path from any gateway to all nodes of the cluster.
+topology of connections between gateways. But there must always be a path
+from any gateway to all nodes of the cluster, otherwise a key won't
+be able to be stored from, or retrieved from some nodes.
 
 It's best to avoid there being multiple paths to a node that go via
 different gateways, since all paths will be tried in parallel when eg,
 uploading a key to the cluster.
+
+A breakdown in communication between gateways will temporarily split the
+cluster. When communication resumes, some keys may need to be copied to
+additional nodes.
diff --git a/doc/todo/git-annex_proxies.mdwn b/doc/todo/git-annex_proxies.mdwn
index 113777c5f1..0dc0836bc2 100644
--- a/doc/todo/git-annex_proxies.mdwn
+++ b/doc/todo/git-annex_proxies.mdwn
@@ -38,14 +38,6 @@ For June's work on [[design/passthrough_proxy]], remaining todos:
   round-robin amoung remotes, and prefer to avoid using remotes that
   other git-annex processes are currently using.
 
-* When a cluster has multiple gateways, and a key is uploaded via one
-  gateway, that gateway learns about every node where the key is stored.
-  But other gateways do not, they only learn about nodes reached via them
-  where the key is stored. This means that another user, syncing with
-  the other gateway, won't know how many copies exist, or necessarily
-  that the key is in the cluster at all. Should gateways broadcast
-  location change messages to other gateways?
-
 * Optimise proxy speed. See design for ideas.
 
 * Use `sendfile()` to avoid data copying overhead when

Changed pages:

todo/git-annex proxies
Changed by: joeyh
Commit type: git
Date: 16:57:08 06/27/24

updates

diff --git a/doc/clusters.mdwn b/doc/clusters.mdwn
index eef4171d76..4f1062ab7f 100644
--- a/doc/clusters.mdwn
+++ b/doc/clusters.mdwn
@@ -191,3 +191,14 @@ served by the current gateway.
 Notice that remotes for cluster nodes have names indicating the path through
 the cluster used to access them. For example, "AMS-NYC-node3" is accessed via
 the AMS gateway, which then relays to NYC where node3 is located.
+
+## cluster topologies
+
+Clusters can be constructed with any number of gateways, and any internal
+topology of connections between gateways. 
+
+There must always be a path from any gateway to all nodes of the cluster.
+
+It's best to avoid there being multiple paths to a node that go via
+different gateways, since all paths will be tried in parallel when eg,
+uploading a key to the cluster.
diff --git a/doc/todo/git-annex_proxies.mdwn b/doc/todo/git-annex_proxies.mdwn
index 81ef99a4e6..113777c5f1 100644
--- a/doc/todo/git-annex_proxies.mdwn
+++ b/doc/todo/git-annex_proxies.mdwn
@@ -29,11 +29,23 @@ For June's work on [[design/passthrough_proxy]], remaining todos:
 * Since proxying to special remotes is not supported yet, and won't be for
   the first release, make it fail in a reasonable way.
 
+* Proxying for special remotes.
+
+* Encryption and chunking. See design for issues.
+
 * Getting a key from a cluster currently always selects the lowest cost
   remote, and always the same remote if cost is the same. Should
   round-robin amoung remotes, and prefer to avoid using remotes that
   other git-annex processes are currently using.
 
+* When a cluster has multiple gateways, and a key is uploaded via one
+  gateway, that gateway learns about every node where the key is stored.
+  But other gateways do not, they only learn about nodes reached via them
+  where the key is stored. This means that another user, syncing with
+  the other gateway, won't know how many copies exist, or necessarily
+  that the key is in the cluster at all. Should gateways broadcast
+  location change messages to other gateways?
+
 * Optimise proxy speed. See design for ideas.
 
 * Use `sendfile()` to avoid data copying overhead when
@@ -41,8 +53,6 @@ For June's work on [[design/passthrough_proxy]], remaining todos:
   Library to use:
   <https://hackage.haskell.org/package/hsyscall-0.4/docs/System-Syscall.html>
 
-* Encryption and chunking. See design for issues.
-
 * Indirect uploads (to be considered). See design.
 
 * Support using a proxy when its url is a P2P address.
@@ -51,7 +61,6 @@ For June's work on [[design/passthrough_proxy]], remaining todos:
 * `viconfig` support for setting preferred content, group, 
   and description of clusters
 
-
 # completed items for June's work on [[design/passthrough_proxy]]:
 
 * UUID discovery via git-annex branch. Add a log file listing UUIDs

Changed pages:

todo/git-annex proxies
Changed by: joeyh
Commit type: git
Date: 16:20:22 06/27/24

distributed cluster cycle prevention
Added BYPASS to P2P protocol, and use it to avoid cycling between
cluster gateways.
Distributed clusters are working well now!

diff --git a/Annex/Cluster.hs b/Annex/Cluster.hs
index 8438881829..fa83170835 100644
--- a/Annex/Cluster.hs
+++ b/Annex/Cluster.hs
@@ -5,7 +5,7 @@
  - Licensed under the GNU AGPL version 3 or higher.
  -}
 
-{-# LANGUAGE RankNTypes #-}
+{-# LANGUAGE RankNTypes, OverloadedStrings #-}
 
 module Annex.Cluster where
 
@@ -17,6 +17,7 @@ import P2P.Proxy
 import P2P.Protocol
 import P2P.IO
 import Annex.Proxy
+import Annex.UUID
 import Logs.Location
 import Logs.PreferredContent
 import Types.Command
@@ -50,24 +51,40 @@ proxyCluster clusteruuid proxydone servermode clientside protoerrhandler = do
 		-- determine. Instead, pick the newest protocol version
 		-- that we and the client both speak. The proxy code
 		-- checks protocol versions when operating on multiple
-		-- nodes.
+		-- nodes, and allows nodes to have different protocol
+		-- versions.
 		let protocolversion = min maxProtocolVersion clientmaxversion
-		selectnode <- clusterProxySelector clusteruuid protocolversion
+		sendClientProtocolVersion clientside othermsg protocolversion
+			(getclientbypass protocolversion) protoerrhandler
+	withclientversion Nothing = proxydone
+
+	getclientbypass protocolversion othermsg =
+		getClientBypass clientside protocolversion othermsg
+			(withclientbypass protocolversion) protoerrhandler
+
+	withclientbypass protocolversion (bypassuuids, othermsg) = do
+		selectnode <- clusterProxySelector clusteruuid protocolversion bypassuuids
 		concurrencyconfig <- getConcurrencyConfig
 		proxy proxydone proxymethods servermode clientside 
 			(fromClusterUUID clusteruuid)
 			selectnode concurrencyconfig protocolversion
 			othermsg protoerrhandler
-	withclientversion Nothing = proxydone
 
-clusterProxySelector :: ClusterUUID -> ProtocolVersion -> Annex ProxySelector
-clusterProxySelector clusteruuid protocolversion = do
+clusterProxySelector :: ClusterUUID -> ProtocolVersion -> Bypass -> Annex ProxySelector
+clusterProxySelector clusteruuid protocolversion (Bypass bypass) = do
 	nodeuuids <- (fromMaybe S.empty . M.lookup clusteruuid . clusterUUIDs)
 		<$> getClusters
-	clusternames <- annexClusters <$> Annex.getGitConfig
+	myclusters <- annexClusters <$> Annex.getGitConfig
 	allremotes <- remoteList
-	let clusterremotes = filter (isnode allremotes nodeuuids clusternames) allremotes
-	nodes <- mapM (proxySshRemoteSide protocolversion) clusterremotes
+	hereu <- getUUID
+	let bypass' = S.insert hereu bypass
+	let clusterremotes = filter (isnode bypass' allremotes nodeuuids myclusters) allremotes
+	fastDebug "Annex.Cluster" $ unwords
+		[ "cluster gateway at", fromUUID hereu
+		, "connecting to", show (map Remote.name clusterremotes)
+		, "bypass", show (S.toList bypass)
+		]
+	nodes <- mapM (proxySshRemoteSide protocolversion (Bypass bypass')) clusterremotes
 	return $ ProxySelector
 		{ proxyCHECKPRESENT = nodecontaining nodes
 		, proxyGET = nodecontaining nodes
@@ -95,27 +112,37 @@ clusterProxySelector clusteruuid protocolversion = do
 		}
   where
 	-- Nodes of the cluster have remote.name.annex-cluster-node
-	-- containing its name. Or they are proxied by a remote
-	-- that has remote.name.annex-cluster-gateway
-	-- containing the cluster's UUID.
-	isnode rs nodeuuids clusternames r = 
+	-- containing its name. 
+	--
+	-- Or, a node can be the cluster proxied by another gateway.
+	isnode bypass' rs nodeuuids myclusters r = 
 		case remoteAnnexClusterNode (Remote.gitconfig r) of
 			Just names
-				| any (isclustername clusternames) names ->
+				| any (isclustername myclusters) names ->
 					flip S.member nodeuuids $ 
 						ClusterNodeUUID $ Remote.uuid r
 				| otherwise -> False
-			Nothing -> case remoteAnnexProxiedBy (Remote.gitconfig r) of
-				Just proxyuuid
-					| Remote.uuid r /= fromClusterUUID clusteruuid -> 
+			Nothing -> isclusterviagateway bypass' rs r
+	
+	-- Is this remote the same cluster, proxied via another gateway?
+	--
+	-- Must avoid bypassed gateways to prevent cycles.
+	isclusterviagateway bypass' rs r = 
+		case mkClusterUUID (Remote.uuid r) of
+			Just cu | cu == clusteruuid ->
+				case remoteAnnexProxiedBy (Remote.gitconfig r) of
+					Just proxyuuid | proxyuuid `S.notMember` bypass' ->
 						not $ null $
-							filter (== clusteruuid) $
-							concatMap (remoteAnnexClusterGateway . Remote.gitconfig) $
+							filter isclustergateway $
 							filter (\p -> Remote.uuid p == proxyuuid) rs
-				_ -> False
+					_ -> False
+			_ -> False
 	
-	isclustername clusternames name = 
-		M.lookup name clusternames == Just clusteruuid
+	isclustergateway r = any (== clusteruuid) $ 
+		remoteAnnexClusterGateway $ Remote.gitconfig r
+
+	isclustername myclusters name = 
+		M.lookup name myclusters == Just clusteruuid
 	
 	nodecontaining nodes k = do
 		locs <- S.fromList <$> loggedLocations k
diff --git a/Annex/Proxy.hs b/Annex/Proxy.hs
index 6eacc04df8..747f393d6d 100644
--- a/Annex/Proxy.hs
+++ b/Annex/Proxy.hs
@@ -15,13 +15,14 @@ import qualified Remote
 import Remote.Helper.Ssh (openP2PShellConnection', closeP2PShellConnection)
 
 -- FIXME: Support special remotes.
-proxySshRemoteSide :: ProtocolVersion -> Remote -> Annex RemoteSide
-proxySshRemoteSide clientmaxversion remote = mkRemoteSide (Remote.uuid remote) $
-	openP2PShellConnection' remote clientmaxversion >>= \case
-		Just conn@(OpenConnection (remoterunst, remoteconn, _)) ->
-			return $ Just 
-				( remoterunst
-				, remoteconn
-				, void $ liftIO $ closeP2PShellConnection conn
-				)
-		_  -> return Nothing
+proxySshRemoteSide :: ProtocolVersion -> Bypass -> Remote -> Annex RemoteSide
+proxySshRemoteSide clientmaxversion bypass remote = 
+	mkRemoteSide (Remote.uuid remote) $
+		openP2PShellConnection' remote clientmaxversion bypass >>= \case
+			Just conn@(OpenConnection (remoterunst, remoteconn, _)) ->
+				return $ Just 
+					( remoterunst
+					, remoteconn
+					, void $ liftIO $ closeP2PShellConnection conn
+					)
+			_  -> return Nothing
diff --git a/Command/P2PStdIO.hs b/Command/P2PStdIO.hs
index 3724d4222a..f72a1d314f 100644
--- a/Command/P2PStdIO.hs
+++ b/Command/P2PStdIO.hs
@@ -67,7 +67,7 @@ performProxy clientuuid servermode remote = do
 		p2pErrHandler
   where
 	withclientversion clientside (Just (clientmaxversion, othermsg)) = do
-		remoteside <- proxySshRemoteSide clientmaxversion remote
+		remoteside <- proxySshRemoteSide clientmaxversion mempty remote
 		protocolversion <- either (const (min P2P.maxProtocolVersion clientmaxversion)) id
 			<$> runRemoteSide remoteside 
 				(P2P.net P2P.getProtocolVersion)
@@ -75,11 +75,14 @@ performProxy clientuuid servermode remote = do
 			closeRemoteSide remoteside
 			p2pDone
 		concurrencyconfig <- noConcurrencyConfig
-		proxy closer proxymethods servermode clientside
+		let runproxy othermsg' = proxy closer proxymethods
+			servermode clientside
 			(Remote.uuid remote)
 			(singleProxySelector remoteside)
 			concurrencyconfig
-			protocolversion othermsg p2pErrHandler
+			protocolversion othermsg' p2pErrHandler
+		sendClientProtocolVersion clientside othermsg protocolversion
+			runproxy p2pErrHandler
 	withclientversion _ Nothing = p2pDone
 	
 	proxymethods = ProxyMethods
diff --git a/P2P/Protocol.hs b/P2P/Protocol.hs
index ae652370f5..a77cf76536 100644
--- a/P2P/Protocol.hs
+++ b/P2P/Protocol.hs
@@ -9,6 +9,7 @@
 
 {-# LANGUAGE DeriveFunctor, TemplateHaskell, FlexibleContexts #-}
 {-# LANGUAGE TypeSynonymInstances, FlexibleInstances, RankNTypes #-}
+{-# LANGUAGE GeneralizedNewtypeDeriving #-}
 {-# OPTIONS_GHC -fno-warn-orphans #-}
 
 module P2P.Protocol where
@@ -37,6 +38,7 @@ import System.IO
 import qualified System.FilePath.ByteString as P
 import qualified Data.ByteString as B
 import qualified Data.ByteString.Lazy as L
+import qualified Data.Set as S
 import Data.Char
 import Control.Applicative
 import Prelude

(Diff truncated)

Changed pages:

forum/Calculate HMAC of special remote
Changed by: lykos
Commit type: web
Date: 10:47:43 06/27/24

diff --git a/doc/forum/Calculate_HMAC_of_special_remote.mdwn b/doc/forum/Calculate_HMAC_of_special_remote.mdwn
new file mode 100644
index 0000000000..f4b2761bff
--- /dev/null
+++ b/doc/forum/Calculate_HMAC_of_special_remote.mdwn
@@ -0,0 +1,14 @@
+Hello,
+
+I need to do some plumbing in a special remote. What is the best way to get the filenames of all the chunks belonging to a key?
+From https://git-annex.branchable.com/internals/ I know that `.log.cnk` stores size and number of chunks. And that part of the cipher in remote.log is used as the HMAC encryption key. 
+
+I found that I can use `openssl` to calculate the HMAC, but what exactly do I need to feed it?
+
+```
+echo $PAYLOAD | openssl sha512 -hex -mac HMAC -macopt hexkey:$KEY_HEX
+```
+
+Or is there a git-annex plumbing command that I overlooked?
+
+--lykos153

Changed pages:

todo/git-annex proxies
Changed by: joeyh
Commit type: git
Date: 19:27:16 06/26/24

avoid loop between cluster gateways
The VIA extension is still needed to avoid some extra work and ugly
messages, but this is enough that it actually works.
This filters out the RemoteSides that are a proxied connection via a
remote gateway to the cluster.
The VIA extension will not filter those out, but will send VIA to them
on connect, which will cause the ones that are accessed via the listed
gateways to be filtered out.

diff --git a/Annex/Cluster.hs b/Annex/Cluster.hs
index 6ddf9029c1..8438881829 100644
--- a/Annex/Cluster.hs
+++ b/Annex/Cluster.hs
@@ -96,8 +96,8 @@ clusterProxySelector clusteruuid protocolversion = do
   where
 	-- Nodes of the cluster have remote.name.annex-cluster-node
 	-- containing its name. Or they are proxied by a remote
-	-- that has remote.name.annex-cluster-node containing the cluster's
-	-- UUID.
+	-- that has remote.name.annex-cluster-gateway
+	-- containing the cluster's UUID.
 	isnode rs nodeuuids clusternames r = 
 		case remoteAnnexClusterNode (Remote.gitconfig r) of
 			Just names
@@ -106,11 +106,13 @@ clusterProxySelector clusteruuid protocolversion = do
 						ClusterNodeUUID $ Remote.uuid r
 				| otherwise -> False
 			Nothing -> case remoteAnnexProxiedBy (Remote.gitconfig r) of
-				Just proxyuuid -> not $ null $
-					filter (== clusteruuid) $
-					concatMap (remoteAnnexClusterGateway . Remote.gitconfig) $
-					filter (\p -> Remote.uuid p == proxyuuid) rs
-				Nothing -> False
+				Just proxyuuid
+					| Remote.uuid r /= fromClusterUUID clusteruuid -> 
+						not $ null $
+							filter (== clusteruuid) $
+							concatMap (remoteAnnexClusterGateway . Remote.gitconfig) $
+							filter (\p -> Remote.uuid p == proxyuuid) rs
+				_ -> False
 	
 	isclustername clusternames name = 
 		M.lookup name clusternames == Just clusteruuid
diff --git a/doc/todo/git-annex_proxies.mdwn b/doc/todo/git-annex_proxies.mdwn
index ac106adceb..206991c731 100644
--- a/doc/todo/git-annex_proxies.mdwn
+++ b/doc/todo/git-annex_proxies.mdwn
@@ -33,6 +33,9 @@ For June's work on [[design/passthrough_proxy]], remaining todos:
   protocol messages on to any remotes that have the same UUID as
   the cluster. Needs VIA extension to P2P protocol to avoid cycles.
 
+  Status: works, but needs VIA extension to avoid ugly messages and extra
+  work
+
 * Getting a key from a cluster currently always selects the lowest cost
   remote, and always the same remote if cost is the same. Should
   round-robin amoung remotes, and prefer to avoid using remotes that

Changed pages:

todo/git-annex proxies
Changed by: joeyh
Commit type: git
Date: 19:02:10 06/26/24

support multi-gateway clusters
VIA extension still needed otherwise a copy to a cluster can loop
forever.

diff --git a/Annex/Cluster.hs b/Annex/Cluster.hs
index 599dc1c417..6ddf9029c1 100644
--- a/Annex/Cluster.hs
+++ b/Annex/Cluster.hs
@@ -65,8 +65,9 @@ clusterProxySelector clusteruuid protocolversion = do
 	nodeuuids <- (fromMaybe S.empty . M.lookup clusteruuid . clusterUUIDs)
 		<$> getClusters
 	clusternames <- annexClusters <$> Annex.getGitConfig
-	remotes <- filter (isnode nodeuuids clusternames) <$> remoteList
-	nodes <- mapM (proxySshRemoteSide protocolversion) remotes
+	allremotes <- remoteList
+	let clusterremotes = filter (isnode allremotes nodeuuids clusternames) allremotes
+	nodes <- mapM (proxySshRemoteSide protocolversion) clusterremotes
 	return $ ProxySelector
 		{ proxyCHECKPRESENT = nodecontaining nodes
 		, proxyGET = nodecontaining nodes
@@ -94,15 +95,22 @@ clusterProxySelector clusteruuid protocolversion = do
 		}
   where
 	-- Nodes of the cluster have remote.name.annex-cluster-node
-	-- containing its name.
-	isnode nodeuuids clusternames r = 
+	-- containing its name. Or they are proxied by a remote
+	-- that has remote.name.annex-cluster-node containing the cluster's
+	-- UUID.
+	isnode rs nodeuuids clusternames r = 
 		case remoteAnnexClusterNode (Remote.gitconfig r) of
-			Nothing -> False
 			Just names
 				| any (isclustername clusternames) names ->
 					flip S.member nodeuuids $ 
 						ClusterNodeUUID $ Remote.uuid r
 				| otherwise -> False
+			Nothing -> case remoteAnnexProxiedBy (Remote.gitconfig r) of
+				Just proxyuuid -> not $ null $
+					filter (== clusteruuid) $
+					concatMap (remoteAnnexClusterGateway . Remote.gitconfig) $
+					filter (\p -> Remote.uuid p == proxyuuid) rs
+				Nothing -> False
 	
 	isclustername clusternames name = 
 		M.lookup name clusternames == Just clusteruuid
diff --git a/doc/todo/git-annex_proxies.mdwn b/doc/todo/git-annex_proxies.mdwn
index a3584840cf..ac106adceb 100644
--- a/doc/todo/git-annex_proxies.mdwn
+++ b/doc/todo/git-annex_proxies.mdwn
@@ -33,12 +33,6 @@ For June's work on [[design/passthrough_proxy]], remaining todos:
   protocol messages on to any remotes that have the same UUID as
   the cluster. Needs VIA extension to P2P protocol to avoid cycles.
 
-  Current status: Distributed cluster nodes are visible,
-  and can be accessed directly, but trying to GET from a cluster
-  fails when the content is located behind a remote gateway.
-  And PUT only sends to the immediate nodes
-  of the cluster, not on to other gateways.
-
 * Getting a key from a cluster currently always selects the lowest cost
   remote, and always the same remote if cost is the same. Should
   round-robin amoung remotes, and prefer to avoid using remotes that

Changed pages:

clusters
Changed by: joeyh
Commit type: git
Date: 18:21:35 06/26/24

update for multi-gateway clusters

diff --git a/doc/clusters.mdwn b/doc/clusters.mdwn
index deb7113f1f..eef4171d76 100644
--- a/doc/clusters.mdwn
+++ b/doc/clusters.mdwn
@@ -15,8 +15,7 @@ remote is added the same as any other remote:
     git remote add bigserver me@bigserver:annex
 
 The gateway publishes information about the cluster to the git-annex
-branch. (See below for how that is configured.) So you may need to fetch
-from it to learn about the cluster:
+branch. So you may need to fetch from it to learn about the cluster:
 
     git fetch bigserver
 
@@ -41,7 +40,7 @@ at once, very efficiently.
     
     $ git-annex whereis bar
 	whereis bar (3 copies)
-	  	acae2ff6-6c1e-8bec-b8b9-397a3755f397 -- my cluster [bigserver-mycluster]
+	  	acae2ff6-6c1e-8bec-b8b9-397a3755f397 -- [bigserver-mycluster]
 	   	9f514001-6dc0-4d83-9af3-c64c96626892 -- node 1 [bigserver-node1]
 	   	d81e0b28-612e-4d73-a4e6-6dabbb03aba1 -- node 2 [bigserver-node2]
 	    5657baca-2f11-11ef-ae1a-5b68c6321dd9 -- node 3 [bigserver-node3]
@@ -56,7 +55,32 @@ clusters.
 A cluster is not a git repository, and so `git pull bigserver-mycluster`
 will not work.
 
-## configuring a cluster
+## preferred content of clusters
+
+The preferred content of the cluster can be configured. This tells
+users what files the cluster as a whole should contain.
+
+To configure the preferred content of a cluster, as well as other related
+things like [[groups|git-annex-group]] and [[required_content]], it's easiest
+to do the configuration in a repository that has the cluster as a remote.
+
+For example:
+
+	$ git-annex wanted bigserver-mycluster standard
+	$ git-annex group bigserver-mycluster archive
+
+By default, when a file is uploaded to a cluster, it is stored on every node of
+the cluster. To control which nodes to store to, the [[preferred_content]] of
+each node can be configured.
+
+It's also a good idea to configure the preferred content of the cluster's
+gateway. To avoid files redundantly being stored on the gateway
+(which remember, is not a node of the cluster), you might make it not want
+any files:
+
+    $ git-annex wanted bigserver nothing
+
+## setting up a cluster
 
 A new cluster first needs to be initialized. Run [[git-annex-initcluster]] in
 the repository that will serve as the cluster's gateway. In the example above,
@@ -74,14 +98,14 @@ In the example above, the three cluster nodes were configured like this:
 	$ git remote add node1 /media/disk1/repo
 	$ git remote add node2 /media/disk2/repo
 	$ git remote add node3 /media/disk3/repo
-	$ git config remote.node1.annex-cluster-node true
-	$ git config remote.node2.annex-cluster-node true
-	$ git config remote.node3.annex-cluster-node true
+	$ git config remote.node1.annex-cluster-node mycluster
+	$ git config remote.node2.annex-cluster-node mycluster
+	$ git config remote.node3.annex-cluster-node mycluster
 
 Finally, run `git-annex updatecluster` to record the cluster configuration
 in the git-annex branch. That tells other repositories about the cluster.
 	
-	$ git-annex updatecluster mycluster
+	$ git-annex updatecluster
 	Added node node1 to cluster: mycluster
 	Added node node2 to cluster: mycluster
 	Added node node3 to cluster: mycluster
@@ -96,27 +120,74 @@ on more than one at a time will likely be faster.
 
     $ git config annex.jobs cpus
 
-## preferred content of clusters
+## adding additional gateways to a cluster
 
-The preferred content of the cluster can be configured. This tells
-users what files the cluster as a whole should contain.
+A cluster can have more than one gateway. One way to use this is to
+make a cluster that is distributed across several locations.
 
-To configure the preferred content of a cluster, as well as other related
-things like [[groups|git-annex-group]] and [[required_content]], it's easiest
-to do the configuration in a repository that has the cluster as a remote.
+Suppose you have a datacenter in AMS, and one in NYC. There
+will be a gateway in each datacenter which provides access to the nodes
+there. And the gateways will relay data between each other as well.
 
-For example:
+Start by setting up the cluster in Amsterdam. The process is the same
+as in the previous section.
 
-	$ git-annex wanted bigserver-mycluster standard
-	$ git-annex group bigserver-mycluster archive
+	AMS$ git-annex initcluster mycluster
+	AMS$ git remote add node1 /media/disk1/repo
+	AMS$ git remote add node2 /media/disk2/repo
+	AMS$ git config remote.node1.annex-cluster-node mycluster
+	AMS$ git config remote.node2.annex-cluster-node mycluster
+	AMS$ git-annex updatecluster
+    AMS$ git config annex.jobs cpus
 
-By default, when a file is uploaded to a cluster, it is stored on every node of
-the cluster. To control which nodes to store to, the [[preferred_content]] of
-each node can be configured.
+Now in a clone of the same repository in NYC, add AMS as a git remote
+accessed with ssh:
 
-It's also a good idea to configure the preferred content of the cluster's
-gateway. To avoid files redundantly being stored on the gateway
-(which remember, is not a node of the cluster), you might make it not want
-any files:
+    NYC$ git remote add AMS me@amsterdam.example.com:annex
+    NYC$ git fetch AMS
 
-    $ git-annex wanted bigserver nothing
+Setting up the cluster in NYC is different, rather than using
+`git-annex initcluster` again (which would make a new, different
+cluster), we ask git-annex to extend the cluster from AMS:
+
+    NYC$ git-annex extendcluster AMS mycluster
+
+The rest of the setup process for NYC is the same, of course different
+nodes are added.
+	
+	NYC$ git remote add node3 /media/disk3/repo
+	NYC$ git remote add node4 /media/disk4/repo
+	NYC$ git config remote.node3.annex-cluster-node mycluster
+	NYC$ git config remote.node4.annex-cluster-node mycluster
+	NYC$ git-annex updatecluster
+    NYC$ git config annex.jobs cpus
+
+Finally, the AMS side of the cluster has to be updated, adding a git remote
+for NYC, and extending the cluster to there as well:
+
+    AMS$ git remote add NYC me@nyc.example.com:annex
+    AMS$ git-annex sync NYC
+    NYC$ git-annex extendcluster NYC mycluster
+    AMS$ git-annex updatecluster
+
+A user can now add either AMS or NYC as a remote, and will have access
+to the entire cluster as either `AMS-mycluster` or `NYC-mycluster`.
+
+    user$ git-annex move foo --to AMS-mycluster
+    move foo (to AMS-mycluster...) ok
+
+Looking at where files end up, all the nodes are visible, not only those
+served by the current gateway.
+
+    user$ git-annex whereis foo
+	whereis foo (4 copies)
+	  	acfc1cb2-b8d5-8393-b8dc-4a419ea38183 -- cluster mycluster [AMS-mycluster]
+	   	11ab09a9-7448-45bd-ab81-3997780d00b3 -- node4 [AMS-NYC-node4]
+	   	36197d0e-6d49-4213-8440-71cbb121e670 -- node2 [AMS-node2]
+	   	43652651-1efa-442a-8333-eb346db31553 -- node3 [AMS-NYC-node3]
+	   	7fb5a77b-77a3-4032-b3e5-536698e308b3 -- node1 [AMS-node1]
+	ok
+
+Notice that remotes for cluster nodes have names indicating the path through
+the cluster used to access them. For example, "AMS-NYC-node3" is accessed via
+the AMS gateway, which then relays to NYC where node3 is located.

Changed pages:

git-annex

todo/git-annex proxies
Changed by: joeyh
Commit type: git
Date: 16:56:16 06/26/24

git-annex-shell: proxy nodes located beyond remote cluster gateways
Walking a tightrope between security and convenience here, because
git-annex-shell needs to only proxy for things when there has been
an explicit, local action to configure them.
In this case, the user has to have run `git-annex extendcluster`,
which now sets annex-cluster-gateway on the remote.
Note that any repositories that the gateway is recorded to
proxy for will be proxied onward. This is not limited to cluster nodes,
because checking the node log would not add any security; someone could
add any uuid to it. The gateway of course then does its own
checking to determine if it will allow proxying for the remote.

diff --git a/CmdLine/GitAnnexShell.hs b/CmdLine/GitAnnexShell.hs
index cc4fb406bc..6f7456bb51 100644
--- a/CmdLine/GitAnnexShell.hs
+++ b/CmdLine/GitAnnexShell.hs
@@ -206,17 +206,29 @@ checkProxy remoteuuid ouruuid = M.lookup ouruuid <$> getProxies >>= \case
 		rs <- concat . byCost <$> remoteList
 		myclusters <- annexClusters <$> Annex.getGitConfig
 		let sameuuid r = uuid r == remoteuuid
-		-- Only proxy for a remote when the git configuration
-		-- allows it.
-		let proxyconfigured r = remoteAnnexProxy (R.gitconfig r)
-			|| (any (`M.member` myclusters) $ fromMaybe [] $ remoteAnnexClusterNode $ R.gitconfig r)
 		let samename r p = name r == proxyRemoteName p
-		case headMaybe (filter (\r -> sameuuid r && proxyconfigured r && any (samename r) ps) rs) of
+		case headMaybe (filter (\r -> sameuuid r && proxyisconfigured rs myclusters r && any (samename r) ps) rs) of
 			Nothing -> notconfigured
 			Just r -> do
 				Annex.changeState $ \st ->
 					st { Annex.proxyremote = Just (Right r) }
 				return True
+	
+	-- Only proxy for a remote when the git configuration
+	-- allows it. This is important to prevent changes to 
+	-- the git-annex branch making git-annex-shell unexpectedly
+	-- proxy for remotes.
+	proxyisconfigured rs myclusters r
+		| remoteAnnexProxy (R.gitconfig r) = True
+		-- Proxy for remotes that are configured as cluster nodes.
+		| any (`M.member` myclusters) (fromMaybe [] $ remoteAnnexClusterNode $ R.gitconfig r) = True
+		-- Proxy for a remote when it is proxied by another remote
+		-- which is itself configured as a cluster gateway.
+		| otherwise = case remoteAnnexProxiedBy (R.gitconfig r) of
+			Just proxyuuid -> not $ null $ 
+				concatMap (remoteAnnexClusterGateway . R.gitconfig) $
+					filter (\p -> R.uuid p == proxyuuid) rs
+			Nothing -> False
 
 	proxyforcluster cu = do
 		clusters <- getClusters
diff --git a/Command/ExtendCluster.hs b/Command/ExtendCluster.hs
index c83877b05a..6fa248d57a 100644
--- a/Command/ExtendCluster.hs
+++ b/Command/ExtendCluster.hs
@@ -13,6 +13,7 @@ import Command
 import qualified Annex
 import Types.Cluster
 import Config
+import Types.GitConfig
 import qualified Remote
 
 import qualified Data.Map as M
@@ -23,11 +24,13 @@ cmd = command "extendcluster" SectionSetup "add an gateway to a cluster"
 
 seek :: CmdParams -> CommandSeek
 seek (remotename:clustername:[]) = Remote.byName (Just clusterremotename) >>= \case
-	Just clusterremote -> 
-		case mkClusterUUID (Remote.uuid clusterremote) of
-			Just cu -> commandAction $ start cu clustername
-			Nothing -> giveup $ clusterremotename 
-				++ " is not a cluster remote."
+	Just clusterremote -> Remote.byName (Just remotename) >>= \case
+		Just gatewayremote -> 
+			case mkClusterUUID (Remote.uuid clusterremote) of
+				Just cu -> commandAction $ start cu clustername gatewayremote
+				Nothing -> giveup $ clusterremotename 
+					++ " is not a cluster remote."
+		Nothing -> giveup $ "No remote named " ++ remotename ++ " exists."
 	Nothing -> giveup $ "Expected to find a cluster remote named " 
 		++ clusterremotename
 		++ " that is accessed via " ++ remotename
@@ -38,12 +41,14 @@ seek (remotename:clustername:[]) = Remote.byName (Just clusterremotename) >>= \c
 	clusterremotename = remotename ++ "-" ++ clustername
 seek _ = giveup "Expected two parameters, gateway and clustername."
 
-start :: ClusterUUID -> String -> CommandStart
-start cu clustername = starting "extendcluster" ai si $ do
+start :: ClusterUUID -> String -> Remote -> CommandStart
+start cu clustername gatewayremote = starting "extendcluster" ai si $ do
 	myclusters <- annexClusters <$> Annex.getGitConfig
+	let setcus f = setConfig f (fromUUID (fromClusterUUID cu))
 	unless (M.member clustername myclusters) $ do
-		setConfig (annexConfig ("cluster." <> encodeBS clustername))
-			(fromUUID (fromClusterUUID cu))
+		setcus $ annexConfig ("cluster." <> encodeBS clustername)
+	setcus $ remoteAnnexConfig gatewayremote $ 
+		remoteGitConfigKey ClusterGatewayField
 	next $ return True
   where
 	ai = ActionItemOther (Just (UnquotedString clustername))
diff --git a/Command/UpdateCluster.hs b/Command/UpdateCluster.hs
index 72b59233a6..f16318bed0 100644
--- a/Command/UpdateCluster.hs
+++ b/Command/UpdateCluster.hs
@@ -80,5 +80,5 @@ findProxiedClusterNodes recordednodes =
   where
 	isproxynode r = 
 		asclusternode r `S.member` recordednodes
-			&& remoteAnnexProxied (R.gitconfig r)
+			&& isJust (remoteAnnexProxiedBy (R.gitconfig r))
 	asclusternode = ClusterNodeUUID . R.uuid
diff --git a/Command/UpdateProxy.hs b/Command/UpdateProxy.hs
index cbe7ae9a81..d21b7321c4 100644
--- a/Command/UpdateProxy.hs
+++ b/Command/UpdateProxy.hs
@@ -83,7 +83,7 @@ findRemoteProxiedClusterNodes = do
 		<$> Annex.getGitConfig
 	clusternodes <- clusterNodeUUIDs <$> getClusters
 	let isproxiedclusternode r
-		| remoteAnnexProxied (R.gitconfig r) =
+		| isJust (remoteAnnexProxiedBy (R.gitconfig r)) =
 			case M.lookup (ClusterNodeUUID (R.uuid r)) clusternodes of
 				Nothing -> False
 				Just s -> not $ S.null $ 
diff --git a/Remote.hs b/Remote.hs
index 9038d2a767..eea052e254 100644
--- a/Remote.hs
+++ b/Remote.hs
@@ -455,7 +455,7 @@ gitSyncableRemote :: Remote -> Bool
 gitSyncableRemote r
 	| gitSyncableRemoteType (remotetype r) 
 		&& isJust (remoteUrl (gitconfig r)) =
-			not (remoteAnnexProxied (gitconfig r))
+			not (isJust (remoteAnnexProxiedBy (gitconfig r)))
 	| otherwise = case remoteUrl (gitconfig r) of
 		Just u | "annex::" `isPrefixOf` u -> True
 		_ -> False
diff --git a/Remote/Git.hs b/Remote/Git.hs
index 6c29c28cfe..89e0da38c1 100644
--- a/Remote/Git.hs
+++ b/Remote/Git.hs
@@ -794,21 +794,22 @@ listProxied proxies rs = concat <$> mapM go rs
 			then pure []
 			else case M.lookup cu proxies of
 				Nothing -> pure []
-				Just s -> catMaybes
-					<$> mapM (mkproxied g r s) (S.toList s)
+				Just proxied -> catMaybes
+					<$> mapM (mkproxied g r gc proxied)
+						(S.toList proxied)
 	
 	proxiedremotename r p = do
 		n <- Git.remoteName r
 		pure $ n ++ "-" ++ proxyRemoteName p
 
-	mkproxied g r proxied p = case proxiedremotename r p of
+	mkproxied g r gc proxied p = case proxiedremotename r p of
 		Nothing -> pure Nothing
-		Just proxyname -> mkproxied' g r proxied p proxyname
+		Just proxyname -> mkproxied' g r gc proxied p proxyname
 	
 	-- The proxied remote is constructed by renaming the proxy remote,
 	-- changing its uuid, and setting the proxied remote's inherited
 	-- configs and uuid in Annex state.
-	mkproxied' g r proxied p proxyname
+	mkproxied' g r gc proxied p proxyname
 		| any isconfig (M.keys (Git.config g)) = pure Nothing
 		| otherwise = do
 			clusters <- getClustersWith id
@@ -830,7 +831,7 @@ listProxied proxies rs = concat <$> mapM go rs
 		annexconfigadjuster clusters r' = 
 			let c = adduuid (configRepoUUID renamedr) $
 				addurl $
-				addproxied $
+				addproxiedby $
 				adjustclusternode clusters $
 				inheritconfigs $ Git.fullconfig r'
 			in r'
@@ -844,7 +845,10 @@ listProxied proxies rs = concat <$> mapM go rs
 		addurl = M.insert (remoteConfig renamedr (remoteGitConfigKey UrlField))
 			[Git.ConfigValue $ encodeBS $ Git.repoLocation r]
 		
-		addproxied = addremoteannexfield ProxiedField True
+		addproxiedby = case remoteAnnexUUID gc of
+			Just u -> addremoteannexfield ProxiedByField
+				[Git.ConfigValue $ fromUUID u]
+			Nothing -> id
 		
 		-- A node of a cluster that is being proxied along with
 		-- that cluster does not need to be synced with
@@ -854,14 +858,14 @@ listProxied proxies rs = concat <$> mapM go rs
 			case M.lookup (ClusterNodeUUID (proxyRemoteUUID p)) (clusterNodeUUIDs clusters) of
 				Just cs
 					| any (\c -> S.member (fromClusterUUID c) proxieduuids) (S.toList cs) ->
-						addremoteannexfield SyncField False
+						addremoteannexfield SyncField
+							[Git.ConfigValue $ Git.Config.boolConfig' False]
 				_ -> id
 
 		proxieduuids = S.map proxyRemoteUUID proxied
 
-		addremoteannexfield f b = M.insert
+		addremoteannexfield f = M.insert
 			(remoteAnnexConfig renamedr (remoteGitConfigKey f))
-			[Git.ConfigValue $ Git.Config.boolConfig' b]
 
 		inheritconfigs c = foldl' inheritconfig c proxyInheritedFields
 		
diff --git a/Remote/Helper/Git.hs b/Remote/Helper/Git.hs
index b240ee0a2f..c37d286df4 100644
--- a/Remote/Helper/Git.hs

(Diff truncated)

Changed pages: