Recent changes to this wiki:

policy on AI generated content
diff --git a/doc/contribute.mdwn b/doc/contribute.mdwn
index a005e22e81..54bf3b8f1c 100644
--- a/doc/contribute.mdwn
+++ b/doc/contribute.mdwn
@@ -61,6 +61,9 @@ or work on porting libraries needed by the Windows port.
 To send patches, either include the patch in a [[bug|bugs]] report (small
 patch) or put up a branch in a git repository containing your changes.
 
+[Policy on AI generated content](https://joeyh.name/blog/entry/policy_on_adding_AI_generated_content_to_my_software_projects/)
+(Summary: It's welcome! But see the page for important details.)
+
 ## learning some Haskell
 
 Want to learn some Haskell to get hacking on git-annex?

update
diff --git a/doc/design/balanced_preferred_content.mdwn b/doc/design/balanced_preferred_content.mdwn
index 2a84b1273c..0e2bc5d417 100644
--- a/doc/design/balanced_preferred_content.mdwn
+++ b/doc/design/balanced_preferred_content.mdwn
@@ -1,4 +1,4 @@
-Say we have 2 backup drives and want to fill them both evenly with files,
+Say we have 2 drives and want to fill them both evenly with files,
 different files in each drive. Currently, preferred content cannot express
 that entirely:
 
@@ -6,11 +6,20 @@ that entirely:
 * Or, can let both repos take whatever files, perhaps at random, that the
   other repo is not know to contain, but then repos will race and both get
   the same file, or similarly if they are not communicating frequently.
+  Existing preferred content expressions such as the one for archive group
+  have this problem.
 
 So, let's add a new expression: `balanced(group)`
 
+## implementation
+
 This would work by taking the list of uuids of all repositories in the
-group, and sorting them, which yields a list from 0..M-1 repositories.
+group that have enough free space to store a key, and sorting them, 
+which yields a list from 0..M-1 repositories.
+
+(To know if a repo has enough free space to store a key 
+will need [[todo/track_free_space_in_repos_via_git-annex_branch]]
+to be implemented.)
 
 To decide which repository wants key K, convert K to a number N in some
 stable way and then `N mod M` yields the number of the repository that
@@ -19,60 +28,29 @@ wants it, while all the rest don't.
 (Since git-annex keys can be pretty long and not all of them are random
 hashes, let's md5sum the key and then use the md5 as a number.)
 
-This expression is stable as long as the members of the group don't change.
-I think that's stable enough to work as a preferred content expression.
+## stability
+
+Note that this preferred content expression will not be stable. A change in 
+the members of the group will change which repository is selected. And
+changes in how full repositories are will also change which repo is
+selected.
 
-Now, you may want to be able to add a third repo and have the data be
-rebalanced, with some moving to it. And that would happen. However, as this
-scheme stands, it's equally likely that adding repo3 will make repo1 and
-repo2 want to swap files between them. So, we'll want to add some
-precautions to avoid a lot of data moving around in this case:
+Without stability, when another repo is added to the group, all data will
+be rebalanced, with some moving to it. Which could be desirable in some
+situations, but the problem is that it's likely that adding repo3 will make
+repo1 and repo2 want to swap some files between them,
+
+So, we'll want to add some precautions to avoid a lot of data moving around
+in such a case:
 
 	((balanced(backup) and not (copies=backup:1)) or present
 
 So once file lands on a backup drive, it stays there, even if more backup
 drives change the balancing.
 
------
-
-Some limitations:
-
-* The item size is not taken into account. One repo could end up with a
-  much larger item or items and so fill up faster. And the other repo
-  wouldn't then notice it was full and take up some slack.
-* With the complicated expression above, adding a new repo when one 
-  is full would not necessarily result in new files going to one of the 2
-  repos that still have space. Some items would end up going to the full
-  repo.
-
-These can be dealt with by noticing when a repo is full and moving some
-of it's files (any will do) to other repos in its group. I don't see a way
-to make preferred content express that movement though; it would need to be
-a manual/scripted process.
-
-> Could the size of each repo be recorded (either actual disk size or
-> desired max size) and when a repo is too full to hold an object, be left
-> out of the set of repos used to calculate where to store that object?
->
-> With the preferred content expression above with "present" in it, 
-> a repo being full would not cause any content to be moved off of it,
-> only new content that had not yet reached any of the repos in the 
-> group would be affected. That seems good.
-> 
-> This would need only a single one-time write to the git-annex branch,
-> to record the repo size. Then update a local counter for each repository
-> from the git-annex branch location log changes. 
-> There is a todo about doing this,
-> [[todo/track_free_space_in_repos_via_git-annex_branch]].
-> 
-> Of course, in the time after the git-annex branch was updated and before
-> it reaches the local repo, a repo can be full without us knowing about
-> it. Stores to it would fail, and perhaps be retried, until the updated
-> git-annex branch was synced.
-
------
-
-What if we have 5 backup repos and want each file to land in 3 of them?
+## use case: 3 of 5
+
+What if we have 5 backup repos and want each key to be stored in 3 of them?
 There's a simple change that can support that:
 `balanced(group:3)`
 
@@ -80,29 +58,15 @@ This works the same as before, but rather than just `N mod M`, take
 `N+I mod M` where I is [0..2] to get the list of 3 repositories that want a
 key.
 
-This does not really avoid the limitations above, but having more repos
-that want each file will reduce the chances that no repo will be able to
-take a given file. In the [[iabackup]] scenario, new clients will just be
-assigned until all the files reach the desired level or replication.
+However, once 3 of those 5 repos get full, new keys will only be able to be
+stored on 2 of them. At that point one or more new repos will need to be
+added to reach the goal of each key being stored in 3 of them. It would be
+possible to rebalance the 3 full repos by moving some keys from them to the
+other 2 repos, and eke out more storage before needing to add new
+repositories. A separate rebalancing pass, that does not use preferred
+content alone, could be implemented to handle this (see below).
 
-However.. Imagine there are 9 repos, all full, and some files have not
-reached desired level of replication. Seems like adding 1 more repo will make
-only 3 in 10 files be wanted by that new repo. Even if the repo has space
-for all the files, it won't be sufficient, and more repos would need to be
-added.
-
-One way to avoid this problem would be if the preferred content was only
-used for the initial distribution of files to a repo. If the repo has
-gotten all the files it wants, it could make a second pass and
-opportunistically get files it doesn't want but that it has space for
-and that don't have enough copies yet.
-Although this gets back to the original problem of multiple repos racing
-downloads and files getting more than the desired number of copies.
-
-> With the above idea of tracking when repos are full, the new repo
-> would want all files when the other 9 repos are full.
-
-----
+## use case: geographically distinct datacenters
 
 Of course this is not limited to backup drives. A more complicated example:
 There are 4 geographically distributed datacenters, each of which has some
@@ -112,7 +76,7 @@ on some drive there.
 This can be implemented by making a group for each datacenter, which all of
 its drives are in, and using `balanced()` to pick the drive that holds the
 copy of the file. The preferred content expression would be eg:
-	
+
     ((balanced(datacenterA) and not (copies=datacenterA:1)) or present
 
 In such a situation, to avoid a `N^2` remote interconnect, there might be a
@@ -126,20 +90,60 @@ the place that `balanced()` picks for a group. Eg,
 `balancedgroup=datacenterA` for 1 copy and `balancedgroup=group:datacenterA:2`
 for N copies.
 
-----
+The [[design/passthrough_proxy]] idea is an alternate way to put a
+repository in front of such a cluster, that does not need additional
+extensions to preferred content.
+
+## split brain situations
+
+Of course, in the time after the git-annex branch was updated and before
+it reaches the local repo, a repo can be full without us knowing about
+it. Stores to it would fail, and perhaps be retried, until the updated
+git-annex branch was synced. 
+
+In the worst case, a split brain situation
+can make the balanced preferred content expression
+pick a different repository to hold two independent
+stores of the same key. Eg, when one side thinks one repo is full,
+and the other side thinks the other repo is full.
+
+If `present` is used in the preferred content, both of them will then
+want to contain it. (Is `present` really needed like shown in the examples
+above?)
+
+If it's not, one of them will drop it and the other will
+usually maintain its copy. It would perhaps be possible for both of
+them to drop it, leading to a re-upload cycle. This needs some research
+to see if it's a real problem. 
+See [[todo/proving_preferred_content_behavior]].
+
+## rebalancing
+
+In both the 3 of 5 use case and a split brain situation, it's possible for
+content to end up not optimally balanced between repositories. git-annex
+can be made to operate in a mode where it does additional work to rebalance
+repositories. 
+
+This can be an option like --rebalance, that changes how the preferred content
+expression is evaluated. The user can choose where and when to run that.
+Eg, it might be run on a node inside a cluster after adding more storage to
+the cluster.
+

(Diff truncated)
Added a comment: Need for git-annex-remote-rclone
diff --git a/doc/special_remotes/rclone/comment_1_3fa761ad2e7a87e160b6d0e86814801c._comment b/doc/special_remotes/rclone/comment_1_3fa761ad2e7a87e160b6d0e86814801c._comment
new file mode 100644
index 0000000000..c698e2154a
--- /dev/null
+++ b/doc/special_remotes/rclone/comment_1_3fa761ad2e7a87e160b6d0e86814801c._comment
@@ -0,0 +1,10 @@
+[[!comment format=mdwn
+ username="oadams"
+ avatar="http://cdn.libravatar.org/avatar/ac166a5f89f10c4108e5150015e6751b"
+ subject="Need for git-annex-remote-rclone"
+ date="2024-03-14T01:03:46Z"
+ content="""
+In order to use rclone as a special remote, the user needs to download a separate Bash scriptfrom https://github.com/DanielDent/git-annex-remote-rclone and put it in their PATH. Since that extra dependency is only a few hundred lines of Bash, I would be interested in attempting to implement `Remote/Rclone.hs` so that the rclone special remote is entirely built into git-annex. However, I wanted to run it by you before more seriously considering investing time in doing that. What are your thoughts on this? I'm assuming the only reason rclone support isn't built into git-annex is just a lack of time and incentive, rather than a more fundamental technical reason. Is that right?
+
+Thanks for all your work on this tool. 
+"""]]

update
diff --git a/doc/design/passthrough_proxy.mdwn b/doc/design/passthrough_proxy.mdwn
index c5c9385168..91830f41ce 100644
--- a/doc/design/passthrough_proxy.mdwn
+++ b/doc/design/passthrough_proxy.mdwn
@@ -117,8 +117,11 @@ the client seems like the best choice by far.
 
 But, git-annex's git remotes don't currently ever do encryption. And
 special remotes don't communicate via the P2P protocol with a git remote.
-So none of git-annex existing remote implementations would be able to handle
-this case. So something will need to be changed in the remote
-implementation to handle this case.
+So none of git-annex's existing remote implementations would be able to handle
+this case. Something will need to be changed in the remote
+implementation for this.
 
 (Chunking has the same problem.)
+
+There's potentially a layering problem here, because exactly how encryption
+(or chunking) works can vary depending on the type of special remote.

update
diff --git a/doc/design/balanced_preferred_content.mdwn b/doc/design/balanced_preferred_content.mdwn
index f46eac8470..2a84b1273c 100644
--- a/doc/design/balanced_preferred_content.mdwn
+++ b/doc/design/balanced_preferred_content.mdwn
@@ -140,4 +140,6 @@ would surely have been in vain.)
 
 ## see also
 
-[[todo/proving_preferred_content_behavior]]
+[[todo/proving_preferred_content_behavior]]  
+[[todo/passthrough_proxy]]  
+
diff --git a/doc/design/passthrough_proxy.mdwn b/doc/design/passthrough_proxy.mdwn
index 16b5dc7f0e..c5c9385168 100644
--- a/doc/design/passthrough_proxy.mdwn
+++ b/doc/design/passthrough_proxy.mdwn
@@ -107,3 +107,18 @@ content. Eg, analize what files are typically requested, and store another
 copy of those on the proxy. Perhaps prioritize storing smaller files, where
 latency tends to swamp transfer speed.
 
+## encryption
+
+When the proxy is in front of a special remote that uses encryption, where
+does the encryption happen? It could either happen on the client before
+sending to the proxy, or the proxy could do the encryption since it
+communicates with the special remote. For security, doing the encryption on
+the client seems like the best choice by far.
+
+But, git-annex's git remotes don't currently ever do encryption. And
+special remotes don't communicate via the P2P protocol with a git remote.
+So none of git-annex existing remote implementations would be able to handle
+this case. So something will need to be changed in the remote
+implementation to handle this case.
+
+(Chunking has the same problem.)

link
diff --git a/doc/design/balanced_preferred_content.mdwn b/doc/design/balanced_preferred_content.mdwn
index 9adf201bcd..f46eac8470 100644
--- a/doc/design/balanced_preferred_content.mdwn
+++ b/doc/design/balanced_preferred_content.mdwn
@@ -137,3 +137,7 @@ In a split brain situation, there would be sets of repos doing work toward
 different solutions. On merge it would make sense to calculate a new
 solution that takes that work into account as well as possible. (Some work
 would surely have been in vain.)
+
+## see also
+
+[[todo/proving_preferred_content_behavior]]

fix spelling
diff --git a/doc/design/passthrouh_proxy.mdwn b/doc/design/passthrough_proxy.mdwn
similarity index 100%
rename from doc/design/passthrouh_proxy.mdwn
rename to doc/design/passthrough_proxy.mdwn

todo
diff --git a/doc/todo/proving_preferred_content_behavior.mdwn b/doc/todo/proving_preferred_content_behavior.mdwn
new file mode 100644
index 0000000000..73f181bc0d
--- /dev/null
+++ b/doc/todo/proving_preferred_content_behavior.mdwn
@@ -0,0 +1,75 @@
+Preferred content expressions can be complicated to write and reason about.
+A complex expression can involve lots of repositories that can get into
+different states, and needs to be written to avoid unwanted behavior.
+
+It would be very handy to provide some way to prove things about behavior
+of preferred content expressions, or a way to simulate the behavior of a
+network of git-annex repositories with a given preferred content configuration 
+
+## motivating examples
+
+For example, consider two reposities A and B. A is in group M and B is in
+group N. A has preferred content `not inallgroup=N` and B has `not inallgroup=M`.
+
+If A contains a file, then B will want to also get a copy. And things
+stabilize there. But if the file is removed from A, then B also wants to
+remove it. And once B has removed it, A wants a copy of it. And then B also
+wants a copy of it. So the result is that the file got transferred twice,
+to end up right back where we started.
+
+The worst case of this is `not present`, where the file gets dropped and
+transferred over and over again. The docs warn against using that one. But
+they can't warn about every bad preferred content expression.
+
+## balanced preferred content
+
+When [[design/balanced_preferred_content]] is added, a whole new level of
+complexity will exist in preferred content expressions, because now an
+expression does not make a file be wanted by a single repository, but
+shards the files amoung repositories in a group. 
+
+And up until this point preferred content expressions have behaved the same no
+matter the sizes of the underlying repositories, but balanced preferred
+content does take repository fullness into account, which further
+complicates fully understanding the behavior.
+
+Notice that `balanced()` (in the current design) is not stable when used
+on its own, and has to be used as part of a larger expression to make it
+stable, eg:
+
+    ((balanced(backup) and not (copies=backup:1)) or present
+
+So perhaps `balanced()` should include the other checks in it,
+to avoid the user shooting themselves in the foot. On the other 
+hand, if `balanced()` implicitly contains `present`, then `not balanced()`
+would include `not present`, which is bad!
+
+(For that matter, what does `not balanced()` even do currently?)
+
+## proof
+
+What could be proved about a preferred content expression?
+
+No idea really. Would be interesting to consider what formal methods can
+do here. Could a SAT solver be used somehow for example?
+
+## static analysis
+
+Clearly `not present` is an problematic preferred content expression. It
+would be good if git-annex warned and/or refused to set such an expression
+if it could detect it. Similarly `not groupwanted` could be detected as a
+problem when the group's preferred content expression contains `present`.
+
+Is there is a more general purpose and not expensive way to detect such
+problematic expressions, that can find problems such as the 
+`not inallgroup=N` example above?
+
+## simulation
+
+Simulation seems fairly straightforward, just simulate the network of
+git-annex repositories with random files with different sizes and
+metadata. Be sure to enforce invariants like numcopies the same as
+git-annex does.
+
+Since users can write preferred content expressions, this should be
+targeted at being used by end users.

update
diff --git a/doc/design/passthrouh_proxy.mdwn b/doc/design/passthrouh_proxy.mdwn
index 7860d0f21f..16b5dc7f0e 100644
--- a/doc/design/passthrouh_proxy.mdwn
+++ b/doc/design/passthrouh_proxy.mdwn
@@ -34,15 +34,15 @@ A proxy would not hold the content of files itself. It would be a clone of
 the git repository though, probably. Uploads and downloads would stream
 through the proxy. The git-annex [[P2P_protocol]] could be relayed in this way. 
 
-## discovering UUIDS
+## UUID discovery
 
-A significant difficulty in implementing a proxy for the P2P protocol is
-that each git-annex remote has a single UUID. But the remote that points at
-the proxy can't just have the UUID of the proxy's repository, git-annex
-needs to know that the remote can be used to access repositories with every
-UUID in the cluster.
+A significant difficulty in implementing a proxy is that each git-annex
+remote has a single UUID. But the remote that points at the proxy can't
+just have the UUID of the proxy's repository, git-annex needs to know that
+the proxy's remote can be used to access repositories with every UUID in
+the cluster.
 
-----
+### UUID discovery via P2P protocol extension
 
 Could the P2P protocol be extended to let the proxy communicate the UUIDs
 of all the repositories behind it?
@@ -57,7 +57,7 @@ a proxy. Then it could do UUID discovery each time git-annex starts up.
 But that adds significant overhead, git-annex would be making a connection
 to the proxy in situations where it is not going to use it.
 
-----
+### UUID discovery via git-annex branch
 
 Could the proxy's set of UUIDs instead be recorded somewhere in the
 git-annex branch?

update
diff --git a/doc/design/passthrouh_proxy.mdwn b/doc/design/passthrouh_proxy.mdwn
index 6b5063e7df..7860d0f21f 100644
--- a/doc/design/passthrouh_proxy.mdwn
+++ b/doc/design/passthrouh_proxy.mdwn
@@ -88,3 +88,22 @@ The remote interface operates on object files stored on disk. See
 [[todo/transitive_transfers]] for discussion of that problem. If proxies
 get implemented, that problem should be revisited.
 
+## speed
+
+A passthrough proxy should be as fast as possible so as not to add overhead
+to a file retrieve, store, or checkpresent. This probably means that
+it keeps TCP connections open to each host in the cluster. It might use a
+protocol with less overhead than ssh.
+
+In the case of checkpresent, it would be possible for the proxy to not
+communicate with the cluster to check that the data is still present on it.
+As long as all access is intermediated via the proxy, its git-annex branch
+could be relied on to always be correct, in theory. Proving that theory,
+making sure to account for all possible race conditions and other scenarios,
+would be necessary for such an optimisation.
+
+Another way the proxy could speed things up is to cache some subset of
+content. Eg, analize what files are typically requested, and store another
+copy of those on the proxy. Perhaps prioritize storing smaller files, where
+latency tends to swamp transfer speed.
+

todo
diff --git a/doc/design/passthrouh_proxy.mdwn b/doc/design/passthrouh_proxy.mdwn
new file mode 100644
index 0000000000..6b5063e7df
--- /dev/null
+++ b/doc/design/passthrouh_proxy.mdwn
@@ -0,0 +1,90 @@
+When [[balanced_preferred_content]] is used, there may be many repositories
+in a location -- either a server or a cluster -- and getting any given file
+may need to access any of them. Configuring remotes for each repository
+adds a lot of complexity, both in setting up access controls on each
+server, and for the user. 
+
+Particularly on the user side, when ssh is used they may have to deal with
+many different ssh host keys, as well as adding new remotes or removing
+existing remotes to keep up with changes are made on the server side.
+
+A proxy would avoid this complexity. It also allows limiting network
+ingress to a single point.
+
+Ideally a proxy would look like any other git-annex remote. All the files
+stored anywhere in the cluster would be available to retrieve from the
+proxy. When a file is sent to the proxy, it would store it somewhere in the
+cluster.
+
+Currently the closest git-annex can get to implementing such a proxy is a
+transfer repository that wants all content that is not yet stored in the
+cluster. This allows incoming transfers to be accepted and distributed to
+nodes of the cluster. To get data back out of the cluster, there has to be
+some communication that it is preferred content (eg, setting metadata),
+then after some delay for it to be copied back to the transfer repository,
+it becomes available for the client to download it. And once it knows the
+client has its copy, it can be removed from the transfer repository.
+
+That is quite slow, and rather clumsy. And it risks the transfer repository
+filling up with data that has been requested by clients that have not yet
+picked it up, or with incoming transfers that have not yet reached the
+cluster.
+
+A proxy would not hold the content of files itself. It would be a clone of
+the git repository though, probably. Uploads and downloads would stream
+through the proxy. The git-annex [[P2P_protocol]] could be relayed in this way. 
+
+## discovering UUIDS
+
+A significant difficulty in implementing a proxy for the P2P protocol is
+that each git-annex remote has a single UUID. But the remote that points at
+the proxy can't just have the UUID of the proxy's repository, git-annex
+needs to know that the remote can be used to access repositories with every
+UUID in the cluster.
+
+----
+
+Could the P2P protocol be extended to let the proxy communicate the UUIDs
+of all the repositories behind it?
+
+Once the client git-annex knows the set of UUIDs behind the proxy, it can
+instantiate a remote object per uuid, each of which accesses the proxy, but
+with a different UUID.
+
+But, git-annx usually only does UUID discovery the first time a ssh remote
+is accessed. So it would need to discover at that point that the remote is
+a proxy. Then it could do UUID discovery each time git-annex starts up.
+But that adds significant overhead, git-annex would be making a connection
+to the proxy in situations where it is not going to use it.
+
+----
+
+Could the proxy's set of UUIDs instead be recorded somewhere in the
+git-annex branch?
+
+With this approach, git-annex would know as soon as it sees the proxy's
+UUID that this is a proxy for this other set of UUIDS. (Unless its
+git-annex branch is not up-to-date.) And then it can instantiate a UUID for
+each remote.
+
+One difficulty with this is that, when the git-annex branch is not up to
+date with changes from the proxy, git-annex may try to access repositories
+that are no longer available behind the proxy. That failure would be
+handled the same as any other currently unavailable repository. Also
+git-annex would not use the full set of repositories, so might not be able
+to store data when eg, all the repositories that is knows about are full.
+Just getting the git-annex back in sync should recover from either
+situation.
+
+## streaming to special remotes
+
+As well as being an intermediary to git-annex repositories, the proxy could
+provide access to other special remotes. That could be an object store like
+S3, which might be internal to the cluster or not. When using a cloud
+service like S3, only the proxy needs to know the access credentials.
+
+Currently git-annex does not support streaming content to special remotes.
+The remote interface operates on object files stored on disk. See
+[[todo/transitive_transfers]] for discussion of that problem. If proxies
+get implemented, that problem should be revisited.
+

thoughts
diff --git a/doc/design/balanced_preferred_content.mdwn b/doc/design/balanced_preferred_content.mdwn
index 0acc3562e3..9adf201bcd 100644
--- a/doc/design/balanced_preferred_content.mdwn
+++ b/doc/design/balanced_preferred_content.mdwn
@@ -104,6 +104,30 @@ downloads and files getting more than the desired number of copies.
 
 ----
 
+Of course this is not limited to backup drives. A more complicated example:
+There are 4 geographically distributed datacenters, each of which has some
+number of drives. Each file should have 1 copy stored in each datacenter,
+on some drive there. 
+
+This can be implemented by making a group for each datacenter, which all of
+its drives are in, and using `balanced()` to pick the drive that holds the
+copy of the file. The preferred content expression would be eg:
+	
+    ((balanced(datacenterA) and not (copies=datacenterA:1)) or present
+
+In such a situation, to avoid a `N^2` remote interconnect, there might be a
+transfer repository in each datacenter, that is in front of its drives. The
+transfer repository should want files that have not yet reached the
+destination drive. How to write a preferred content expression for that?
+It might be sufficient to use `copies=datacenterA:1`, so long as the file
+reaching any drive in the datacenter is enough. But may want to add
+something analagous to `inallgroup=` that checks if a file is in
+the place that `balanced()` picks for a group. Eg, 
+`balancedgroup=datacenterA` for 1 copy and `balancedgroup=group:datacenterA:2`
+for N copies.
+
+----
+
 Another possibility to think about is to have one repo calculate which
 files to store on which repos, to best distribute and pack them. The first
 repo that writes a solution would win and other nodes would work to move

diff --git a/doc/bugs/Get_crashes_when_remote_contains_non-english_chars.mdwn b/doc/bugs/Get_crashes_when_remote_contains_non-english_chars.mdwn
new file mode 100644
index 0000000000..fe6098f754
--- /dev/null
+++ b/doc/bugs/Get_crashes_when_remote_contains_non-english_chars.mdwn
@@ -0,0 +1,107 @@
+Hi,
+
+### Please describe the problem.
+I'm trying to set up a git-annex repo for my books/technical papers to have easy access to them on my desktop and laptop. I'm using a centralized server (following [this guide](https://git-annex.branchable.com/tips/centralized_git_repository_tutorial/on_your_own_server/)) to make it easy to sync between my machines.
+
+The issue is however that sqlite crashes when I'm trying to get a file from my server. See the log further down for the error message. I'm suspecting it is due to the repo on my server is named `Böcker` (swedish name for books). It does work if I'm cloning it locally on my server. E.g.
+
+[[!format sh """
+$ ssh server
+$ git clone /mnt/Valhalla/Böcker books
+$ cd books
+$ git annex init
+init  ok
+(recording state in git...)
+$ git annex get facklitteratur/rapporter/143-rods.pdf
+get facklitteratur/rapporter/143-rods.pdf (from origin...) 
+ok                                
+(recording state in git...)
+"""]]
+
+And if I use the `ssh://server/~/books` as a remote it works fine.
+
+### What steps will reproduce the problem?
+[[!format sh """
+$ ssh server
+server$ mkdir /tmp/Böcker
+server$ cd /tmp/Böcker
+server$ git init
+server$ git annex init
+server$ dd if=/dev/zeo of=foo bs=4k count=1
+server$ git annex add foo
+server$ git commit -m "Add foo"
+server$ exit
+$ git clone ssh://server/tmp/Böcker /tmp/Böcker
+$ cd Böcker
+$ git annex init
+$ git annex get foo
+"""]]
+Note that `foo` need to have some size hence the use of `dd`, just doing `touch foo` did not trigger the issue for me.
+
+### What version of git-annex are you using? On what operating system?
+On my desktop:
+[[!format sh """
+$ git annex version
+git-annex version: 10.20240227
+build flags: Assistant Webapp Pairing Inotify DBus DesktopNotify TorrentParser MagicMime Feeds Testsuite S3 WebDAV
+dependency versions: aws-0.23 bloomfilter-2.0.1.2 cryptonite-0.30 DAV-1.3.4 feed-1.3.2.1 ghc-9.2.5 http-client-0.7.13.1 persistent-sqlite-2.13.1.0 torrent-10000.1.1 uuid-1.3.15 yesod-1.6.2.1
+key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL X*
+remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs httpalso borg hook external
+operating system: linux x86_64
+supported repository versions: 8 9 10
+upgrade supported from repository versions: 0 1 2 3 4 5 6 7 8 9 10
+"""]]
+
+Running Guix with linux kernel 6.6.18
+
+On my server:
+[[!format sh """
+$ git annex version
+git-annex version: 10.20240227-gbee3abab14f99f3e3d981d8255ca0dd4ff124a84
+build flags: Assistant Webapp Pairing Inotify DBus DesktopNotify TorrentParser MagicMime Benchmark Feeds Testsuite S3 WebDAV
+dependency versions: aws-0.24 bloomfilter-2.0.1.2 crypton-0.34 DAV-1.3.4 feed-1.3.2.1 ghc-9.2.8 http-client-0.7.15 persistent-sqlite-2.13.1.0 torrent-10000.1.3 uuid-1.3.15 yesod-1.6.2.1
+key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL X*
+remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs httpalso borg hook external
+operating system: linux x86_64
+supported repository versions: 8 9 10
+upgrade supported from repository versions: 0 1 2 3 4 5 6 7 8 9 10
+local repository version: 10
+"""]]
+Running Arch Linux with kernel 6.7.9-arch1-1
+
+### Please provide any additional information below.
+Here is the error message I get without any changes to the config (I can silence the warning by enabling annex.sshcaching).
+[[!format sh """
+⎣plattfot@desktop Böcker⎦ git annex get facklitteratur/rapporter/143-rods.pdf
+get facklitteratur/rapporter/143-rods.pdf (from origin...) 
+
+  You have enabled concurrency, but git-annex is not able to use ssh connection caching. This may result in multiple ssh processes prompting for passwords at the same time.
+
+  annex.sshcaching is not set to true
+git-annex: sqlite worker thread crashed: SQLite3 returned ErrorCan'tOpen while attempting to perform open "/mnt/Valhalla/B\65533\65533cker/.git/annex/keysdb/db".
+p2pstdio: 1 failed
+
+  Lost connection (fd:19: hGetChar: end of file)
+
+  Transfer failed
+
+  Unable to access these remotes: origin
+
+  Maybe add some of these git remotes (git remote add ...):
+  	8c92de49-1a89-4870-8186-b0099ad84be6 -- plattfot@server:~/books
+failed
+get: 1 failed
+
+# End of transcript or log.
+"""]]
+
+Note that this was run after I tested to create a repo with the name books.
+
+### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
+
+Sadly no, I manage to hit this on my first try of git-annex.  But I have known about git-annex for a few years, never found a good use for it in my workflow. As I've been using Syncthing for bigger files and that has been working ok.  But with that you pretty much get all or nothing and conflicts are no fun to resolve.  With my books I want to have my .bib file in git but somehow sync the files. I did not want to put everything in a normal git repo as the books directory takes about 64GB. After looking around I remembered git-annex and reading the walkthrough it seems to be a perfect fit! The killer feature in my opinion is that I can specify what files to sync. Which will be handy on my laptop as I don't really want to use up 64GB just so I can read one or two papers.
+
+I'm not giving up on this that easily. Worst case I'll just rename my repo on my server to Books.
+
+Thank you for all the hours developing this software!
+

comment
diff --git a/doc/bugs/git-annex_is_slow_at_reading_file_content/comment_13_0eb12582a3e182b697a07b833cfbe384._comment b/doc/bugs/git-annex_is_slow_at_reading_file_content/comment_13_0eb12582a3e182b697a07b833cfbe384._comment
new file mode 100644
index 0000000000..91690fe040
--- /dev/null
+++ b/doc/bugs/git-annex_is_slow_at_reading_file_content/comment_13_0eb12582a3e182b697a07b833cfbe384._comment
@@ -0,0 +1,13 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 13"""
+ date="2024-03-11T13:43:19Z"
+ content="""
+<https://hackage.haskell.org/package/botan-low> is another possibility.
+There is a significant effort ongoing to build up this library stack in
+haskell. It does need the botan C library to be installed separately
+unfortunately (I've suggested they embed it).
+
+I have not benchmarked it yet but the docs say it supports hardware
+accellerated SHA1, SHA2, SHA3 on x86 and also SHA1, SHA2 on arm64.
+"""]]

update
diff --git a/Database/ContentIdentifier.hs b/Database/ContentIdentifier.hs
index d40bf4faf5..bbf67dcfb1 100644
--- a/Database/ContentIdentifier.hs
+++ b/Database/ContentIdentifier.hs
@@ -77,7 +77,7 @@ databaseIsEmpty (ContentIdentifierHandle _ b) = b
 -- ContentIndentifiersKeyRemoteCidIndex speeds up queries like 
 -- getContentIdentifiers, but it is not used for
 -- getContentIdentifierKeys. ContentIndentifiersCidRemoteKeyIndex was
--- added to speed that up.
+-- addedto speed that up.
 share [mkPersist sqlSettings, mkMigrate "migrateContentIdentifier"] [persistLowerCase|
 ContentIdentifiers
   remote UUID
diff --git a/doc/todo/track_free_space_in_repos_via_git-annex_branch.mdwn b/doc/todo/track_free_space_in_repos_via_git-annex_branch.mdwn
index a50c698eb0..3b0acecbcc 100644
--- a/doc/todo/track_free_space_in_repos_via_git-annex_branch.mdwn
+++ b/doc/todo/track_free_space_in_repos_via_git-annex_branch.mdwn
@@ -78,3 +78,11 @@ Note that the use of `git cat-file` in union merge is not --buffer
 streaming, so is slower than the patch parsing method that was discussed in
 the previous section. So it might be possible to speed up git-annex branch
 merging using patch parsing.
+
+Note that Database.ContentIdentifier and Database.ImportFeed also update
+by diffing from the old to new git-annex branch (with `git cat-file` to
+read log files) so could also be sped up by being done at git-annex branch
+merge time. Those are less expensive than diffing the location logs only
+because the logs they diff are less often used, and the work is only 
+done when relevant commands are run.
+

update
diff --git a/doc/design/balanced_preferred_content.mdwn b/doc/design/balanced_preferred_content.mdwn
index 9a3f9badbd..0acc3562e3 100644
--- a/doc/design/balanced_preferred_content.mdwn
+++ b/doc/design/balanced_preferred_content.mdwn
@@ -7,7 +7,7 @@ that entirely:
   other repo is not know to contain, but then repos will race and both get
   the same file, or similarly if they are not communicating frequently.
 
-So, let's add a new expression: `balanced_amoung(group)`
+So, let's add a new expression: `balanced(group)`
 
 This would work by taking the list of uuids of all repositories in the
 group, and sorting them, which yields a list from 0..M-1 repositories.
@@ -28,7 +28,7 @@ scheme stands, it's equally likely that adding repo3 will make repo1 and
 repo2 want to swap files between them. So, we'll want to add some
 precautions to avoid a lot of data moving around in this case:
 
-	((balanced_amoung(backup) and not (copies=backup:1)) or present
+	((balanced(backup) and not (copies=backup:1)) or present
 
 So once file lands on a backup drive, it stays there, even if more backup
 drives change the balancing.
@@ -74,7 +74,7 @@ a manual/scripted process.
 
 What if we have 5 backup repos and want each file to land in 3 of them?
 There's a simple change that can support that:
-`balanced_amoung(group:3)`
+`balanced(group:3)`
 
 This works the same as before, but rather than just `N mod M`, take
 `N+I mod M` where I is [0..2] to get the list of 3 repositories that want a
diff --git a/doc/todo/track_free_space_in_repos_via_git-annex_branch.mdwn b/doc/todo/track_free_space_in_repos_via_git-annex_branch.mdwn
index 4cb82798ff..a50c698eb0 100644
--- a/doc/todo/track_free_space_in_repos_via_git-annex_branch.mdwn
+++ b/doc/todo/track_free_space_in_repos_via_git-annex_branch.mdwn
@@ -21,11 +21,7 @@ repos that had a maxsize recorded, essentially for free.
 
 But 8 seconds is rather a long time to block a `git-annex push`
 type command. Which would be needed if any remote's preferred content
-expression used `balanced_amoung`.
-
-It would help some to cache the calculated sizes in eq a sqlite db, update
-the cache after sending or dropping content, and invalidate the cache when
-git-annex branch update merges in a git-annex branch from elsewhere.
+expression used the free space information.
 
 Would it be possible to update incrementally from the previous git-annex
 branch to the current one? That's essentially what `git-annex log
@@ -39,13 +35,46 @@ particular git-annex branch commit. We don't care about sizes at
 intermediate points in time, which that command does calculate.
 
 See [[todo/info_--size-history]] for the subtleties that had to be handled.
-In particular, diffing from the previous git-annex branch commit to current may
+In particular, compating the previous git-annex branch commit to current may
 yield lines that seem to indicate content was added to a repo, but in fact
-that repo already had that content at the previous git-annex branch commit.
-So it seems it would have to look up the location log's value at the 
-previous commit, either querying the git-annex branch or cached state.
+that repo already had that content at the previous git-annex branch commit
+and another log line was recorded elsewhere redundantly.
+So it needs to look at the location log's value at the 
+previous commit in order to determine if a change to a log should be
+counted.
 
 Worst case, that's queries of the location log file for every single key.
 If queried from git, that would be slow -- slower than `git-annex info`'s
 streaming approach. If they were all cached in a sqlite database, it might
 manage to be faster?
+
+## incremental update via git diff
+
+Could `git diff -U1000000` be used and the patch parsed to get the complete
+old and new location log? (Assuming no log file ever reaches a million
+lines.) I tried this in my big repo, and even diffing from the first
+git-annex branch commit to the last took 7.54 seconds. 
+
+Compare that with the method used by `git-annex info`'s size gathering, of
+dumping out the content of all files on the branch with `git ls-tree -r
+git-annex |awk '{print $3}'|git cat-file --batch --buffer`, which only
+takes 3 seconds. So, this is not ideal when diffing to too old a point.
+
+Diffing in my big repo to the git-annex branch from 2020 takes 4 seconds.  
+... from 3 months ago takes 2 seconds.  
+... from 1 week ago takes 1 second.  
+
+## incremental update when merging git-annex branch
+
+When merging git-annex branch changes into .git/annex/index, 
+it already diffs between the branch and the index and uses `git cat-file`
+to get both versions of the file in order to union merge them.
+
+That's essentially the same information needed to do the incremental update
+of the repo sizes. So could update sizes at the same time as merging the
+git-annex branch. That would be essentially free!
+
+Note that the use of `git cat-file` in union merge is not --buffer
+streaming, so is slower than the patch parsing method that was discussed in
+the previous section. So it might be possible to speed up git-annex branch
+merging using patch parsing.

Added a comment: TLS v1.2 EMS (Extended Master/Main Secret)
diff --git a/doc/bugs/tls__58___peer_does_not_support_Extended_Main_Secret/comment_1_731db4542017bef831bc7a06b70f79fb._comment b/doc/bugs/tls__58___peer_does_not_support_Extended_Main_Secret/comment_1_731db4542017bef831bc7a06b70f79fb._comment
new file mode 100644
index 0000000000..88e84961b9
--- /dev/null
+++ b/doc/bugs/tls__58___peer_does_not_support_Extended_Main_Secret/comment_1_731db4542017bef831bc7a06b70f79fb._comment
@@ -0,0 +1,26 @@
+[[!comment format=mdwn
+ username="ewen"
+ avatar="http://cdn.libravatar.org/avatar/605b2981cb52b4af268455dee7a4f64e"
+ subject="TLS v1.2 EMS (Extended Master/Main Secret)"
+ date="2024-03-07T03:01:20Z"
+ content="""
+From some more research it seems that Extended Master Secret (aka Extended Main Secret) is a TLS 1.2 only extension, to work around a problem with TLS 1.2 (eg, [2015 post about the problem](https://www.tripwire.com/state-of-security/tls-extended-master-secret-extension-fixing-a-hole-in-tls)).
+
+TLS v1.3 doesn't have this problem, by design, AFAIK.  And thus clients/servers supporting TLS v1.3 entirely avoids the problem (possibly why I have only found it on a few servers; the one I looked into in detail definitely won't connect with TLS v1.3 right now, but they're looking into it).
+
+The webserver support can be confirmed with, eg forced TLS v1.2:
+
+```
+echo \"\" | openssl s_client -tls1_2 -connect WEBSERVER:443 2>&1 | egrep \"Protocol|Extended master\"
+```
+
+and forced TLS v1.3 to check if that will work:
+
+```
+echo \"\" | openssl s_client -tls1_3 -connect WEBSERVER:443
+```
+
+Hopefully that means the number of impacted sites is *relatively* small (eg, ones that haven't enabled TLS v1.3 support in the last 5+ years).
+
+Ewen
+"""]]

diff --git a/doc/bugs/tls__58___peer_does_not_support_Extended_Main_Secret.mdwn b/doc/bugs/tls__58___peer_does_not_support_Extended_Main_Secret.mdwn
new file mode 100644
index 0000000000..5723e07957
--- /dev/null
+++ b/doc/bugs/tls__58___peer_does_not_support_Extended_Main_Secret.mdwn
@@ -0,0 +1,101 @@
+### Please describe the problem.
+
+On the current [macOS `HomeBrew` build of the current git-annex](https://formulae.brew.sh/formula/git-annex) (10.20240227), it appears that the build dependencies have dragged in the [latest Haskell `tls` 
+package](https://hackage.haskell.org/package/tls-2.0.1/docs/Network-TLS.html).  Which now defaults `supportedExtendedMainSecret` to `RequireEMS` (previously it seems to have been `AllowEMS`; see eg [darcs bug report of similar error](https://bugs.darcs.net/issue2715)).
+
+The result of this is that some podcast feeds, from webservers which do not support EMS, fail with an error, eg:
+
+```
+importfeed https://risky.biz/feeds/risky-business
+  download failed: HandshakeFailed (Error_Protocol "peer does not support Extended Main Secret" HandshakeFailure)
+
+  warning: downloading the feed failed (feed: https://risky.biz/feeds/risky-business)
+ok
+```
+
+(And presumably this will also affect some non-podcast HTTPS downloads; I found it in a podcast download context.)
+
+I believe this "Extended Main Secret" is also known as "Extended Master Secret", aka [RFC 7627](https://www.ietf.org/rfc/rfc7627.html), which was written up in 2015.  So I can understand why ~9 years later the Haskell `tls` library is defaulting to insisting on EMS in a new major version.  Unfortunately not all webservers, especially podcast feed webservers, have caught up with this.
+
+As best I can tell git annex is getting this `tls` dependency via [`http-client`](https://hackage.haskell.org/package/http-client) which uses [`http-client-tls`](https://hackage.haskell.org/package/http-client-tls-0.3.6.3), and `http-client-tls` appears to just have a `tls (>=1.2)` dependency, which is presumably how `tls-2.0.0` / `tls-2.0.1` got dragged in, with these new defaults.
+
+I'm unclear if git-annex is in a position to pass `AllowEMS` to the TLS library (and thus restore to the old default).  But at least in the short term it might be worth considering doing that if possible.
+
+### What steps will reproduce the problem?
+
+Currently I have three podcast feeds (two from the same webserver) which fail:
+
+```
+git annex importfeed https://risky.biz/feeds/risky-business
+```
+
+```
+git annex importfeed https://risky.biz/feeds/risky-business-news
+```
+
+```
+git annex importfeed https://www.thecultureoftech.com/index.php/feed/podcast/ 
+```
+
+(Given the irony that the first two are are an InfoSec podcast, I have also reported this missing EMS extension support to them as well, so it may get fixed before you try it.)
+
+It looks like I've also had one media file download fail repeatedly for the same reason (but the podcast feed itself downloads okay):
+
+```
+git annex addurl https://traffic.omny.fm/d/clips/53b6fe2a-4ef6-4356-ae92-a61500df6da0/40b3f537-c161-4823-ae44-af3a007e121b/b2682900-b36c-447b-812d-b1290049fea8/audio.mp3
+```
+
+### What version of git-annex are you using? On what operating system?
+
+git annex 10.20240227, on macOS Ventura (13.6.3).  With git annex installed from HomeBrew.
+
+```
+ewen@basadi:~$ git annex version
+git-annex version: 10.20240227
+build flags: Assistant Webapp Pairing FsEvents TorrentParser MagicMime Benchmark Feeds Testsuite S3 WebDAV
+dependency versions: aws-0.24.1 bloomfilter-2.0.1.2 crypton-0.34 DAV-1.3.4 feed-1.3.2.1 ghc-9.6.3 http-client-0.7.16 persistent-sqlite-2.13.3.0 torrent-10000.1.3 uuid-1.3.15 yesod-1.6.2.1
+key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL X*
+remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs httpalso borg hook external
+operating system: darwin x86_64
+supported repository versions: 8 9 10
+upgrade supported from repository versions: 0 1 2 3 4 5 6 7 8 9 10
+ewen@basadi:~$ 
+```
+
+### Please provide any additional information below.
+
+[[!format sh """
+ewen@basadi:~/Music/podcasts$ git annex importfeed https://www.thecultureoftech.com/index.php/feed/podcast/ 
+importfeed gathering known urls ok
+importfeed https://www.thecultureoftech.com/index.php/feed/podcast/ 
+  download failed: HandshakeFailed (Error_Protocol "peer does not support Extended Main Secret" HandshakeFailure)
+
+  warning: downloading the feed failed (feed: https://www.thecultureoftech.com/index.php/feed/podcast/)
+ok
+ewen@basadi:~/Music/podcasts$ 
+ewen@basadi:~/Music/podcasts$ git annex addurl https://traffic.omny.fm/d/clips/53b6fe2a-4ef6-4356-ae92-a61500df6da0/40b3f537-c161-4823-ae44-af3a007e121b/b2682900-b36c-447b-812d-b1290049fea8/audio.mp3
+addurl https://traffic.omny.fm/d/clips/53b6fe2a-4ef6-4356-ae92-a61500df6da0/40b3f537-c161-4823-ae44-af3a007e121b/b2682900-b36c-447b-812d-b1290049fea8/audio.mp3 
+git-annex: HttpExceptionRequest Request {
+  host                 = "traffic.omny.fm"
+  port                 = 443
+  secure               = True
+  requestHeaders       = [("Accept-Encoding",""),("User-Agent","git-annex/10.20240227")]
+  path                 = "/d/clips/53b6fe2a-4ef6-4356-ae92-a61500df6da0/40b3f537-c161-4823-ae44-af3a007e121b/b2682900-b36c-447b-812d-b1290049fea8/audio.mp3"
+  queryString          = ""
+  method               = "HEAD"
+  proxy                = Nothing
+  rawBody              = False
+  redirectCount        = 10
+  responseTimeout      = ResponseTimeoutDefault
+  requestVersion       = HTTP/1.1
+  proxySecureMode      = ProxySecureWithConnect
+}
+ (InternalException (HandshakeFailed (Error_Protocol "peer does not support Extended Main Secret" HandshakeFailure)))
+failed
+addurl: 1 failed
+ewen@basadi:~/Music/podcasts$ 
+"""]]
+
+### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
+
+Absolutely, I've been using git-annex as my podcatcher (among other reasons) for about a decade at this point.  Thanks for developing it!

Added a comment
diff --git a/doc/forum/Possible_to_restore_from_an_encrypted_s3_remote__63__/comment_2_ddd6a56d583fd5b9d676365f4ebd9b68._comment b/doc/forum/Possible_to_restore_from_an_encrypted_s3_remote__63__/comment_2_ddd6a56d583fd5b9d676365f4ebd9b68._comment
new file mode 100644
index 0000000000..48c4394cfc
--- /dev/null
+++ b/doc/forum/Possible_to_restore_from_an_encrypted_s3_remote__63__/comment_2_ddd6a56d583fd5b9d676365f4ebd9b68._comment
@@ -0,0 +1,30 @@
+[[!comment format=mdwn
+ username="bbigras"
+ avatar="http://cdn.libravatar.org/avatar/f1c0201e3f1435eaab02c803a33c52ae"
+ subject="comment 2"
+ date="2024-03-06T17:03:50Z"
+ content="""
+I'm not sure I understand what you mean.
+
+Do you mean that I should clone the s3 bucket with a tool like git-remote-s3? I'm not sure how to do that.
+
+I tried downloading the whole bucket from backblaze, but since I enabled encryption on the bucket, backblaze doesn't let me download the encrypted files. I'm not sure yet if I can get those files using their api.
+
+Just in case I didn't explain correctly what I'm trying to do, I'm trying to do something like this, on a new computer with a fresh ~/Document:
+
+```bash
+❯ cd ~/Documents
+❯ git init
+Initialized empty Git repository in /home/bbigras/Documents/.git/
+❯ git annex init
+init  ok
+(recording state in git...)
+❯ git annex initremote backblaze type=S3 signature=v4 host=s3.us-west-000.backblazeb2.com bucket=my-bucket protocol=https encryption=hybrid keyid=my-key-id
+initremote backblaze (encryption setup) (to gpg keys: my-key-id) (checking bucket...)
+  The bucket already exists, and its annex-uuid file indicates it is used by a different special remote.
+
+git-annex: Cannot reuse this bucket.
+failed
+initremote: 1 failed
+```
+"""]]

Added a comment: Still experimental?
diff --git a/doc/tuning/comment_5_0143b798e75ad25c5917794a49a879fb._comment b/doc/tuning/comment_5_0143b798e75ad25c5917794a49a879fb._comment
new file mode 100644
index 0000000000..aee3de42db
--- /dev/null
+++ b/doc/tuning/comment_5_0143b798e75ad25c5917794a49a879fb._comment
@@ -0,0 +1,13 @@
+[[!comment format=mdwn
+ username="imlew"
+ avatar="http://cdn.libravatar.org/avatar/23858c3eed3c3ea9e21522f4c999f1ed"
+ subject="Still experimental?"
+ date="2024-03-06T12:26:56Z"
+ content="""
+`annex.tune.objecthash1=true` and `annex.tune.branchhash1=true` seem like they could be helpful in reducing git annex's inode usage, but the disclaimer about this feature being experimental is a little worrying.
+
+Since this it is over 10 years old though, is it still considered experimental or has it graduated to being a stable feature? I.e. will using this meaningfully increase the chance of losing data?
+
+
+Also, what is the (potential) benefit of using lowercase for the hashes?
+"""]]

add reregisterurl command
What this can currently be used for is only to change an url from being
used by a special remote to being used by the web remote.
This could have been a --move-from option to registerurl. But, that would
have complicated its option and --batch processing, and also would have
complicated unregisterurl, which is implemented on top of
Command.Registerurl. So, a separate command was actually less complicated
to implement.
The generic description of the command is because I want to make this
command a catch-all for other url updating kind of things, if there are
ever any more. Also because it was hard to come up with a good name for the
specific action. I considered `git-annex moveurl`, but that seems to
indicate data is perhaps actually being moved, and seems to sit at the same
level as addurl and rmurl, and this command is at the plumbing
level of registerurl and unregisterurl.
Sponsored-by: Dartmouth College's DANDI project
diff --git a/CHANGELOG b/CHANGELOG
index 2bc030c19d..e7a608e51c 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -5,6 +5,8 @@ git-annex (10.20240228) UNRELEASED; urgency=medium
     annexed files be verified with a checksum that is calculated
     on a later download from the web. This will become the default later.
   * Added dependency on unbounded-delays.
+  * reregisterurl: New command that can change an url from being
+    used by a special remote to being used by the web remote.
 
  -- Joey Hess <id@joeyh.name>  Tue, 27 Feb 2024 13:07:10 -0400
 
diff --git a/CmdLine/GitAnnex.hs b/CmdLine/GitAnnex.hs
index 47a37a3de7..debf30fffd 100644
--- a/CmdLine/GitAnnex.hs
+++ b/CmdLine/GitAnnex.hs
@@ -34,6 +34,7 @@ import qualified Command.MatchExpression
 import qualified Command.FromKey
 import qualified Command.RegisterUrl
 import qualified Command.UnregisterUrl
+import qualified Command.ReregisterUrl
 import qualified Command.SetKey
 import qualified Command.DropKey
 import qualified Command.Transferrer
@@ -196,6 +197,7 @@ cmds testoptparser testrunner mkbenchmarkgenerator = map addGitAnnexCommonOption
 	, Command.FromKey.cmd
 	, Command.RegisterUrl.cmd
 	, Command.UnregisterUrl.cmd
+	, Command.ReregisterUrl.cmd
 	, Command.SetKey.cmd
 	, Command.DropKey.cmd
 	, Command.Transferrer.cmd
diff --git a/Command/ReregisterUrl.hs b/Command/ReregisterUrl.hs
new file mode 100644
index 0000000000..5d95f0da43
--- /dev/null
+++ b/Command/ReregisterUrl.hs
@@ -0,0 +1,79 @@
+{- git-annex command
+ -
+ - Copyright 2015-2024 Joey Hess <id@joeyh.name>
+ -
+ - Licensed under the GNU AGPL version 3 or higher.
+ -}
+
+module Command.ReregisterUrl where
+
+import Command
+import Logs.Web
+import Command.FromKey (keyOpt, keyOpt')
+import qualified Remote
+import Git.Types
+
+cmd :: Command
+cmd = withAnnexOptions [jsonOptions] $ command "reregisterurl"
+	SectionPlumbing "updates url registration information"
+	(paramKey)
+	(seek <$$> optParser)
+
+data ReregisterUrlOptions = ReregisterUrlOptions
+	{ keyOpts :: CmdParams
+	, batchOption :: BatchMode
+	, moveFromOption :: Maybe (DeferredParse Remote)
+	}
+
+optParser :: CmdParamsDesc -> Parser ReregisterUrlOptions
+optParser desc = ReregisterUrlOptions
+	<$> cmdParams desc
+	<*> parseBatchOption False
+	<*> optional (mkParseRemoteOption <$> parseMoveFromOption)
+
+parseMoveFromOption :: Parser RemoteName
+parseMoveFromOption = strOption
+	( long "move-from" <> metavar paramRemote
+	<> completeRemotes
+	)
+
+seek :: ReregisterUrlOptions -> CommandSeek
+seek o = case (batchOption o, keyOpts o) of
+	(Batch fmt, _) -> seekBatch o fmt
+	(NoBatch, ps) -> commandAction (start o ps)
+
+seekBatch :: ReregisterUrlOptions -> BatchFormat -> CommandSeek
+seekBatch o fmt = batchOnly Nothing (keyOpts o) $
+	batchInput fmt (pure . parsebatch) $
+		batchCommandAction . start' o
+  where
+	parsebatch l = case keyOpt' l of
+		Left e -> Left e
+		Right k -> Right k
+
+start :: ReregisterUrlOptions -> [String] -> CommandStart
+start o (keyname:[]) = start' o (si, keyOpt keyname)
+  where
+	si = SeekInput [keyname]
+start _ _ = giveup "specify a key"
+
+start' :: ReregisterUrlOptions -> (SeekInput, Key) -> CommandStart
+start' o (si, key) =
+	starting "reregisterurl" ai si $
+		perform o key
+  where
+	ai = ActionItemKey key
+
+perform :: ReregisterUrlOptions -> Key -> CommandPerform
+perform o key = maybe (pure Nothing) (Just <$$> getParsed) (moveFromOption o) >>= \case
+	Nothing -> next $ return True
+	Just r -> do
+		us <- map fst
+			. filter (\(_, d) -> d == OtherDownloader)
+			. map getDownloader
+			<$> getUrls key
+		us' <- filterM (\u -> (== r) <$> Remote.claimingUrl u) us
+		forM_ us' $ \u -> do
+			setUrlMissing key (setDownloader u OtherDownloader)
+			setUrlPresent key u
+		next $ return True
diff --git a/doc/forum/how_to___34__move__34___URL_between_remotes__63__/comment_6_260c238435f8f5e53511de7e574dd53f._comment b/doc/forum/how_to___34__move__34___URL_between_remotes__63__/comment_6_260c238435f8f5e53511de7e574dd53f._comment
new file mode 100644
index 0000000000..f1114fc3ba
--- /dev/null
+++ b/doc/forum/how_to___34__move__34___URL_between_remotes__63__/comment_6_260c238435f8f5e53511de7e574dd53f._comment
@@ -0,0 +1,29 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 6"""
+ date="2024-03-05T18:11:32Z"
+ content="""
+Went far enough down implementing `registerurl --move-from` to be sure that
+it would complicate the code far more than just adding a new `moveurl`
+command. So despite it being a fairly unusual situation, a new command
+is better than that option.
+
+And implemented it:
+
+	joey@darkstar:~/tmp/x>git-annex registerurl WORM--bar http://example.com/bar.fooregisterurl http://example.com/bar.foo ok
+	joey@darkstar:~/tmp/x> git-annex whereis --key WORM--bar
+	whereis WORM--bar (1 copy)
+	  	dca0b5f9-659a-4928-84db-ff9fd74d8fc8 -- [foo]
+	
+	  foo: http://example.com/bar.foo
+	ok
+	joey@darkstar:~/tmp/x>git-annex reregisterurl WORM--bar --move-from=foo
+	reregisterurl WORM--bar ok
+	joey@darkstar:~/tmp/x>git-annex whereis --key WORM--bar
+	whereis WORM--bar (2 copies)
+	  	00000000-0000-0000-0000-000000000001 -- web
+	   	dca0b5f9-659a-4928-84db-ff9fd74d8fc8 -- [foo]
+	
+	  web: http://example.com/bar.foo
+	ok
+"""]]
diff --git a/doc/git-annex-registerurl.mdwn b/doc/git-annex-registerurl.mdwn
index 753d06b704..bf5133b8db 100644
--- a/doc/git-annex-registerurl.mdwn
+++ b/doc/git-annex-registerurl.mdwn
@@ -68,6 +68,8 @@ special remote that claims it. (Usually the web special remote.)
 
 [[git-annex-unregisterurl]](1)
 
+[[git-annex-reregisterurl]](1)
+
 # AUTHOR
 
 Joey Hess <id@joeyh.name>
diff --git a/doc/git-annex-reregisterurl.mdwn b/doc/git-annex-reregisterurl.mdwn
new file mode 100644
index 0000000000..c22a0c6e98
--- /dev/null
+++ b/doc/git-annex-reregisterurl.mdwn
@@ -0,0 +1,64 @@
+# NAME
+
+git-annex reregisterurl - updates url registration information
+
+# SYNOPSIS
+
+git annex reregisterurl `[key]`
+
+# DESCRIPTION
+
+This plumbing-level command updates information about the urls that are
+registered for a key.
+
+# OPTIONS
+
+* `--move-from=name|uuid`
+
+  For each key, update any urls that are currently claimed by the
+  specified remote to be instead used by the web special remote.
+
+  This could be used eg, when a special remote was needed to provide
+  authorization to get an url, but the url has now become publically
+  available and so the web special remote can be used.
+
+  Note that, like `git-annex unregisterurl`, using this option unregisters
+  an url from a special remote, but it does not mark the content as not
+  present in that special remote. However, like `git-annex registerurl`,
+  this option does mark content as being present in the web special remote.

(Diff truncated)
small problem
diff --git a/doc/todo/migration_to_VURL_by_default.mdwn b/doc/todo/migration_to_VURL_by_default.mdwn
index ce78eb857d..7d1c4b4385 100644
--- a/doc/todo/migration_to_VURL_by_default.mdwn
+++ b/doc/todo/migration_to_VURL_by_default.mdwn
@@ -21,3 +21,9 @@ WORM keys if they really want to.
 However, I don't think there's enough reason to want to use URL keys to add
 configuration of which kind of keys addurl uses, once VURL is the default.
 --[[Joey]]
+
+> One way this might cause trouble is that current `git-annex registerurl`
+> and `unregisterurl` (and `fromkey`)  when passed an url rather than a key,
+> generates an URL key. If that is changed to generate a VURL key, then
+> it might break some workflow, particularly one where an url was
+> registered as an URL key and is now being unregistered.

comment
diff --git a/doc/forum/how_to___34__move__34___URL_between_remotes__63__/comment_5_712f40f27b27dfaf58c416ca7dc19045._comment b/doc/forum/how_to___34__move__34___URL_between_remotes__63__/comment_5_712f40f27b27dfaf58c416ca7dc19045._comment
new file mode 100644
index 0000000000..c9239dc1ae
--- /dev/null
+++ b/doc/forum/how_to___34__move__34___URL_between_remotes__63__/comment_5_712f40f27b27dfaf58c416ca7dc19045._comment
@@ -0,0 +1,17 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 5"""
+ date="2024-03-05T17:23:34Z"
+ content="""
+Thing is that `unregisterurl` does not mark content as not present in a
+special remote. Except for the web which is a special case. Reason is
+that not having an url registered by a special remote does not prevent getting
+content from that special remote in general.
+
+So my idea for `--move-from=foo` was that it should behave the same way
+since it's the same as `registerurl + unregisterurl`.
+
+If you're going to remove/disable the special remote anyway, it won't
+matter whether git-annex thinks it contains content, I suppose? Or you
+could use setpresentkey of course.
+"""]]

add todo for tracking free space in repos via git-annex branch
For balanced preferred content perhaps, or just for git-annex info
display.
Sponsored-by: unqueued on Patreon
diff --git a/doc/design/balanced_preferred_content.mdwn b/doc/design/balanced_preferred_content.mdwn
index 2953eb16a0..9a3f9badbd 100644
--- a/doc/design/balanced_preferred_content.mdwn
+++ b/doc/design/balanced_preferred_content.mdwn
@@ -62,6 +62,8 @@ a manual/scripted process.
 > This would need only a single one-time write to the git-annex branch,
 > to record the repo size. Then update a local counter for each repository
 > from the git-annex branch location log changes. 
+> There is a todo about doing this,
+> [[todo/track_free_space_in_repos_via_git-annex_branch]].
 > 
 > Of course, in the time after the git-annex branch was updated and before
 > it reaches the local repo, a repo can be full without us knowing about
diff --git a/doc/todo/track_free_space_in_repos_via_git-annex_branch.mdwn b/doc/todo/track_free_space_in_repos_via_git-annex_branch.mdwn
new file mode 100644
index 0000000000..4cb82798ff
--- /dev/null
+++ b/doc/todo/track_free_space_in_repos_via_git-annex_branch.mdwn
@@ -0,0 +1,51 @@
+If the total space available in a repository for annex objects is recorded
+on the git-annex branch (by the user running a command probably, or perhaps
+automatically), then it is possible to examine the git-annex branch and
+tell how much free space a remote has available.
+
+One use case is just to display it in `git-annex info`. But a more
+compelling use case is [[design/balanced_preferred_content]], which needs a
+way to tell when an object is too large to store on a repository, so that
+it can be redirected to be stored on another repository in the same group.
+
+This was actually a fairly common feature request early on in git-annex
+and I probably should have thought about it more back then!
+
+`git-annex info` has recently started summing up the sizes of repositories
+from location logs, and is well optimised. In my big repository, that takes
+8.54 seconds of its total runtime.
+
+Since info already knows the repo sizes, just adding a `git-annex maxsize
+here 200gb` type of command would let it display the free space of all
+repos that had a maxsize recorded, essentially for free.
+
+But 8 seconds is rather a long time to block a `git-annex push`
+type command. Which would be needed if any remote's preferred content
+expression used `balanced_amoung`.
+
+It would help some to cache the calculated sizes in eq a sqlite db, update
+the cache after sending or dropping content, and invalidate the cache when
+git-annex branch update merges in a git-annex branch from elsewhere.
+
+Would it be possible to update incrementally from the previous git-annex
+branch to the current one? That's essentially what `git-annex log
+--sizesof` does for each commit on the git-annex branch, so could
+imagine adapting that to store its state on disk, so it can resume
+at a new git-annex branch commit.
+
+Perhaps a less expensive implementation than `git-annex log --sizesof`
+is possible, to get only the current sizes, if the past sizes are known at a
+particular git-annex branch commit. We don't care about sizes at
+intermediate points in time, which that command does calculate.
+
+See [[todo/info_--size-history]] for the subtleties that had to be handled.
+In particular, diffing from the previous git-annex branch commit to current may
+yield lines that seem to indicate content was added to a repo, but in fact
+that repo already had that content at the previous git-annex branch commit.
+So it seems it would have to look up the location log's value at the 
+previous commit, either querying the git-annex branch or cached state.
+
+Worst case, that's queries of the location log file for every single key.
+If queried from git, that would be slow -- slower than `git-annex info`'s
+streaming approach. If they were all cached in a sqlite database, it might
+manage to be faster?

update
diff --git a/doc/thanks/list b/doc/thanks/list
index f64604c81f..4a1178473d 100644
--- a/doc/thanks/list
+++ b/doc/thanks/list
@@ -114,3 +114,5 @@ Alexander Thompson,
 Nathaniel B, 
 kk, 
 Jaime Marquínez Ferrándiz, 
+Stephen Seo, 
+Antoine Balaine, 

Added a comment
diff --git a/doc/submodules/comment_12_d59882712e0547c748d3b354e8607d62._comment b/doc/submodules/comment_12_d59882712e0547c748d3b354e8607d62._comment
new file mode 100644
index 0000000000..d48ebbc83d
--- /dev/null
+++ b/doc/submodules/comment_12_d59882712e0547c748d3b354e8607d62._comment
@@ -0,0 +1,19 @@
+[[!comment format=mdwn
+ username="TTTTAAAx"
+ avatar="http://cdn.libravatar.org/avatar/9edd4b69b9f9fc9b8c1cb8ecd03902d5"
+ subject="comment 12"
+ date="2024-03-05T05:21:59Z"
+ content="""
+I agree with you. Specifically, [this](https://git-annex.branchable.com/todo/detect_and_handle_submodules_after_path_changed_by_mv/) is related to git, not git-annex.
+
+I wanted to run `git annex assist` just and expect to be fixed.
+
+I wrote a simple python script to fix the issue, which changed the path of the submodule. And by adding the script in pre-commit hook, I can do `git annex assist` now without fixing the submodules manually.
+(I used python to manipulate the text files such as dot-gitmodules etc. Perl or Ruby would be much compact in this job.)
+
+It seems to work well for my needs, even though just a prototype script. 
+
+After some polishing (and testing), I'll upload the script here.
+
+Thank you.
+"""]]

response
diff --git a/doc/forum/Possible_to_restore_from_an_encrypted_s3_remote__63__/comment_1_98033f3324bf78af886fa0f003f5eff8._comment b/doc/forum/Possible_to_restore_from_an_encrypted_s3_remote__63__/comment_1_98033f3324bf78af886fa0f003f5eff8._comment
new file mode 100644
index 0000000000..83b3f73c73
--- /dev/null
+++ b/doc/forum/Possible_to_restore_from_an_encrypted_s3_remote__63__/comment_1_98033f3324bf78af886fa0f003f5eff8._comment
@@ -0,0 +1,9 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 1"""
+ date="2024-03-04T21:17:26Z"
+ content="""
+You need to clone to git repository where you initialized the special
+remote the first time, then `git-annex enableremote` will enable using it
+from the clone.
+"""]]

comment
diff --git a/doc/forum/New_external_special_remote_for_rclone/comment_1_8845852338bbb7e46c0d5585d5884faf._comment b/doc/forum/New_external_special_remote_for_rclone/comment_1_8845852338bbb7e46c0d5585d5884faf._comment
new file mode 100644
index 0000000000..fe900d7054
--- /dev/null
+++ b/doc/forum/New_external_special_remote_for_rclone/comment_1_8845852338bbb7e46c0d5585d5884faf._comment
@@ -0,0 +1,15 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 1"""
+ date="2024-03-04T21:10:44Z"
+ content="""
+Cool! Do let us know when it's available and we can add it to the list of
+special remotes.
+
+WRT the simple export interface, there is a PR on git-annex-remote-rclone
+that adds it
+<https://github.com/git-annex-remote-rclone/git-annex-remote-rclone/pull/62>.
+But, it got hung up on a question of avoiding possible data loss if two
+git-annexes are exporting the same file. That sounds like a problem you
+will need to consider in your implementation too.
+"""]]

thoughts
diff --git a/doc/design/balanced_preferred_content.mdwn b/doc/design/balanced_preferred_content.mdwn
index c099642605..2953eb16a0 100644
--- a/doc/design/balanced_preferred_content.mdwn
+++ b/doc/design/balanced_preferred_content.mdwn
@@ -26,7 +26,7 @@ Now, you may want to be able to add a third repo and have the data be
 rebalanced, with some moving to it. And that would happen. However, as this
 scheme stands, it's equally likely that adding repo3 will make repo1 and
 repo2 want to swap files between them. So, we'll want to add some
-precautions to avoid a lof of data moving around in this case:
+precautions to avoid a lot of data moving around in this case:
 
 	((balanced_amoung(backup) and not (copies=backup:1)) or present
 
@@ -50,6 +50,24 @@ of it's files (any will do) to other repos in its group. I don't see a way
 to make preferred content express that movement though; it would need to be
 a manual/scripted process.
 
+> Could the size of each repo be recorded (either actual disk size or
+> desired max size) and when a repo is too full to hold an object, be left
+> out of the set of repos used to calculate where to store that object?
+>
+> With the preferred content expression above with "present" in it, 
+> a repo being full would not cause any content to be moved off of it,
+> only new content that had not yet reached any of the repos in the 
+> group would be affected. That seems good.
+> 
+> This would need only a single one-time write to the git-annex branch,
+> to record the repo size. Then update a local counter for each repository
+> from the git-annex branch location log changes. 
+> 
+> Of course, in the time after the git-annex branch was updated and before
+> it reaches the local repo, a repo can be full without us knowing about
+> it. Stores to it would fail, and perhaps be retried, until the updated
+> git-annex branch was synced.
+
 -----
 
 What if we have 5 backup repos and want each file to land in 3 of them?
@@ -78,3 +96,18 @@ opportunistically get files it doesn't want but that it has space for
 and that don't have enough copies yet.
 Although this gets back to the original problem of multiple repos racing
 downloads and files getting more than the desired number of copies.
+
+> With the above idea of tracking when repos are full, the new repo
+> would want all files when the other 9 repos are full.
+
+----
+
+Another possibility to think about is to have one repo calculate which
+files to store on which repos, to best distribute and pack them. The first
+repo that writes a solution would win and other nodes would work to move
+files around as needed. 
+
+In a split brain situation, there would be sets of repos doing work toward 
+different solutions. On merge it would make sense to calculate a new
+solution that takes that work into account as well as possible. (Some work
+would surely have been in vain.)

diff --git a/doc/forum/New_external_special_remote_for_rclone.mdwn b/doc/forum/New_external_special_remote_for_rclone.mdwn
new file mode 100644
index 0000000000..5ec948db41
--- /dev/null
+++ b/doc/forum/New_external_special_remote_for_rclone.mdwn
@@ -0,0 +1,7 @@
+Hi folks, I'm working on a new external special remote for rclone named "git-annex-remote-rclone-goyle", or "garrgoyle" for short. (I chose a unique name to disambiguate from the preexisting [git-annex-remote-rclone](https://github.com/git-annex-remote-rclone/git-annex-remote-rclone) project.)
+
+Like git-annex-remote-rclone, garrgoyle only supports the [external special remote protocol](https://git-annex.branchable.com/design/external_special_remote_protocol/). In the near future, I'd like to add support for the *simple export interface* (see [export_and_import_appendix](https://git-annex.branchable.com/design/external_special_remote_protocol/export_and_import_appendix/)).
+
+Garrgoyle seems to be faster than git-annex-remote-rclone. Anecdotally, it was 10x faster when I asked to copy ~20 files that were already present on a "drive" remote. I think the difference is that garrgoyle pays the rclone startup cost once, rather than once per action.
+
+I've uploaded a PR to rclone's GitHub repo <https://github.com/rclone/rclone/pull/7654>. If anyone is willing to alpha test, I would greatly appreciate it! Feedback on the PR is also welcome.

diff --git a/doc/forum/Possible_to_restore_from_an_encrypted_s3_remote__63__.mdwn b/doc/forum/Possible_to_restore_from_an_encrypted_s3_remote__63__.mdwn
new file mode 100644
index 0000000000..6502a5ad5d
--- /dev/null
+++ b/doc/forum/Possible_to_restore_from_an_encrypted_s3_remote__63__.mdwn
@@ -0,0 +1,5 @@
+I tried creating a new git annex repo on my desktop and adding the remote but I think I got an error about both repos not having the same idea. I might be wrong.
+
+Is it possible to init a git annex repo on my desktop using an encrypted s3 remote?
+
+thanks

Added a comment
diff --git a/doc/bugs/git-annex_is_slow_at_reading_file_content/comment_12_466a365bca48065cc6b9e280b5c385a5._comment b/doc/bugs/git-annex_is_slow_at_reading_file_content/comment_12_466a365bca48065cc6b9e280b5c385a5._comment
new file mode 100644
index 0000000000..e7078e5a92
--- /dev/null
+++ b/doc/bugs/git-annex_is_slow_at_reading_file_content/comment_12_466a365bca48065cc6b9e280b5c385a5._comment
@@ -0,0 +1,18 @@
+[[!comment format=mdwn
+ username="Atemu"
+ avatar="http://cdn.libravatar.org/avatar/86b8c2d893dfdf2146e1bbb8ac4165fb"
+ subject="comment 12"
+ date="2024-03-02T08:32:40Z"
+ content="""
+Thank you for looking into this again.
+
+> I'd rather avoid OpenSSL wrappers because adding a C library dependency on openssl will complicate building git-annex in some situations.
+
+Would it be possible to make this a build-time option perhaps?
+
+git-annex without SIMD hashing obviously still *works* fast enough for many purposes as its the status quo but having it would be a greatly appreciated optimisation by many. It'd be great to have the option to enable it wherever possible and simply fall back to non-SIMD where it isn't.
+
+> Also, whatever library git-annex uses needs to support incremental hashing, otherwise git-annex has to pay a performance penalty of re-reading a file to hash it after download, rather than hashing while downloading.
+
+Agreed. Incremental hashing is too important to lose over a general optimisation like this.
+"""]]

clarification
diff --git a/doc/git-annex-migrate.mdwn b/doc/git-annex-migrate.mdwn
index 235a32c5e3..89cfeb963f 100644
--- a/doc/git-annex-migrate.mdwn
+++ b/doc/git-annex-migrate.mdwn
@@ -74,7 +74,8 @@ format.
 
   Keys often include the size of their content, which is generally a useful
   thing. In fact, this command defaults to adding missing size information
-  to keys. With this option, the size information is removed instead.
+  to keys in most migrations. With this option, the size information is
+  removed instead.
 
   One use of this option is to convert URL keys that were added
   by `git-annex addurl --fast` to ones that would have been added if

add potential list
diff --git a/doc/projects/dandi/potential.mdwn b/doc/projects/dandi/potential.mdwn
new file mode 100644
index 0000000000..89fe4d579d
--- /dev/null
+++ b/doc/projects/dandi/potential.mdwn
@@ -0,0 +1,5 @@
+These are TODOs that have been tagged as potentially being useful for
+[[/projects/Dandi]] or a related project to fund work on.
+
+[[!inline pages="todo/* and !todo/done and !link(todo/done) and tagged(projects/dandi/potential)"
+sort=mtime feeds=no actions=yes archive=yes show=0 template=buglist]]

implement URL to VURL migration
This needs the content to be present in order to hash it. But it's not
possible for a module used by Backend.URL to call inAnnex because that
would entail a dependency loop. So instead, rely on the fact that
Command.Migrate calls inAnnex before performing a migration.
But, Command.ExamineKey calls fastMigrate and the key may or may not
exist, and it's not wanting to actually perform a migration in any case.
To handle that, had to add an additional value to fastMigrate to
indicate whether the content is inAnnex.
Factored generateEquivilantKey out of Remote.Web.
Note that migrateFromURLToVURL hardcodes use of the SHA256E backend.
It would have been difficult not to, given all the dependency loop
issues. But --backend and annex.backend are used to tell git-annex
migrate to use VURL in any case, so there's no config knob that
the user could expect to configure that.
Sponsored-by: Brock Spratlen on Patreon
diff --git a/Backend.hs b/Backend.hs
index 10f2234a0c..216b59fb4a 100644
--- a/Backend.hs
+++ b/Backend.hs
@@ -38,11 +38,6 @@ import qualified Backend.VURL
 builtinList :: [Backend]
 builtinList = regularBackendList ++ Backend.VURL.backends
 
-{- The default hashing backend. This must use a cryptographically secure
- - hash. -}
-defaultHashBackend :: Backend
-defaultHashBackend = Prelude.head builtinList
-
 {- Backend to use by default when generating a new key. Takes git config
  - and --backend option into account. -}
 defaultBackend :: Annex Backend
diff --git a/Backend/Hash.hs b/Backend/Hash.hs
index 3bf3bde1c5..9768550adf 100644
--- a/Backend/Hash.hs
+++ b/Backend/Hash.hs
@@ -53,8 +53,10 @@ cryptographicallySecure (Blake2spHash _) = True
 cryptographicallySecure SHA1Hash = False
 cryptographicallySecure MD5Hash = False
 
-{- Order is slightly significant; want SHA256 first, and more general
- - sizes earlier. -}
+{- Order is significant. The first hash is the default one that git-annex
+ - uses, and must be cryptographically secure. 
+ -
+ - Also, want more common sizes earlier than uncommon sizes. -}
 hashes :: [Hash]
 hashes = concat 
 	[ map (SHA2Hash . HashSize) [256, 512, 224, 384]
@@ -167,8 +169,8 @@ needsUpgrade key = or
 	, not (hasExt (fromKey keyVariety key)) && keyHash key /= S.fromShort (fromKey keyName key)
 	]
 
-trivialMigrate :: Key -> Backend -> AssociatedFile -> Annex (Maybe Key)
-trivialMigrate oldkey newbackend afile = trivialMigrate' oldkey newbackend afile
+trivialMigrate :: Key -> Backend -> AssociatedFile -> Bool -> Annex (Maybe Key)
+trivialMigrate oldkey newbackend afile _inannex = trivialMigrate' oldkey newbackend afile
 	<$> (annexMaxExtensionLength <$> Annex.getGitConfig)
 
 trivialMigrate' :: Key -> Backend -> AssociatedFile -> Maybe Int -> Maybe Key
diff --git a/Backend/URL.hs b/Backend/URL.hs
index af7427a104..d68b2196e3 100644
--- a/Backend/URL.hs
+++ b/Backend/URL.hs
@@ -14,6 +14,7 @@ import Annex.Common
 import Types.Key
 import Types.Backend
 import Backend.Utilities
+import Backend.VURL.Utilities (migrateFromURLToVURL)
 
 backends :: [Backend]
 backends = [backendURL]
@@ -25,7 +26,7 @@ backendURL = Backend
 	, verifyKeyContent = Nothing
 	, verifyKeyContentIncrementally = Nothing
 	, canUpgradeKey = Nothing
-	, fastMigrate = Nothing
+	, fastMigrate = Just migrateFromURLToVURL
 	-- The content of an url can change at any time, so URL keys are
 	-- not stable.
 	, isStableKey = const False
diff --git a/Backend/VURL.hs b/Backend/VURL.hs
index 9a23967438..d27436805b 100644
--- a/Backend/VURL.hs
+++ b/Backend/VURL.hs
@@ -99,4 +99,3 @@ backendVURL = Backend
 	allowedequiv ek = fromKey keyVariety ek /= VURLKey
 	varietymap = makeVarietyMap regularBackendList
 	getbackend ek = maybeLookupBackendVarietyMap (fromKey keyVariety ek) varietymap
-
diff --git a/Backend/VURL/Utilities.hs b/Backend/VURL/Utilities.hs
new file mode 100644
index 0000000000..1a9a286702
--- /dev/null
+++ b/Backend/VURL/Utilities.hs
@@ -0,0 +1,51 @@
+{- git-annex VURL backend utilities
+ -
+ - Copyright 2024 Joey Hess <id@joeyh.name>
+ -
+ - Licensed under the GNU AGPL version 3 or higher.
+ -}
+
+module Backend.VURL.Utilities where
+
+import Annex.Common
+import Types.Key
+import Types.Backend
+import Types.KeySource
+import Logs.EquivilantKeys
+import qualified Backend.Hash
+import Utility.Metered
+
+migrateFromURLToVURL :: Key -> Backend -> AssociatedFile -> Bool -> Annex (Maybe Key)
+migrateFromURLToVURL oldkey newbackend _af inannex
+	| inannex && fromKey keyVariety oldkey == URLKey && backendVariety newbackend == VURLKey = do
+		let newkey = mkKey $ const $
+			(keyData oldkey)
+				{ keyVariety = VURLKey }
+		contentfile <- calcRepo (gitAnnexLocation oldkey)
+		generateEquivilantKey hashbackend contentfile >>= \case
+			Nothing -> return Nothing
+			Just ek -> do
+				setEquivilantKey newkey ek
+				return (Just newkey)
+	| otherwise = do
+		liftIO $ print ("migrateFromURL", inannex, fromKey keyVariety oldkey)
+		return Nothing
+  where
+	-- Relies on the first hash being cryptographically secure, and the
+	-- default hash used by git-annex.
+	hashbackend = Prelude.head Backend.Hash.backends
+
+-- The Backend must use a cryptographically secure hash.
+generateEquivilantKey :: Backend -> RawFilePath -> Annex (Maybe Key)
+generateEquivilantKey b f =
+	case genKey b of
+		Just genkey -> do
+			showSideAction (UnquotedString Backend.Hash.descChecksum)
+			Just <$> genkey source nullMeterUpdate
+		Nothing -> return Nothing
+  where
+	source = KeySource
+		{ keyFilename = mempty -- avoid adding any extension
+		, contentLocation = f
+		, inodeCache = Nothing
+		}
diff --git a/Backend/Variety.hs b/Backend/Variety.hs
index c0dd924eb0..b4da6f2a96 100644
--- a/Backend/Variety.hs
+++ b/Backend/Variety.hs
@@ -25,6 +25,10 @@ regularBackendList = Backend.Hash.backends
 	++ Backend.WORM.backends 
 	++ Backend.URL.backends
 
+{- The default hashing backend. -}
+defaultHashBackend :: Backend
+defaultHashBackend = Prelude.head regularBackendList
+
 makeVarietyMap :: [Backend] -> M.Map KeyVariety Backend
 makeVarietyMap l = M.fromList $ zip (map backendVariety l) l
 
diff --git a/Backend/WORM.hs b/Backend/WORM.hs
index d936302682..2e2df45004 100644
--- a/Backend/WORM.hs
+++ b/Backend/WORM.hs
@@ -59,8 +59,8 @@ needsUpgrade :: Key -> Bool
 needsUpgrade key =
 	any (`S8.elem` S.fromShort (fromKey keyName key)) [' ', '\r']
 
-removeProblemChars :: Key -> Backend -> AssociatedFile -> Annex (Maybe Key)
-removeProblemChars oldkey newbackend _
+removeProblemChars :: Key -> Backend -> AssociatedFile -> Bool -> Annex (Maybe Key)
+removeProblemChars oldkey newbackend _ _
 	| migratable = return $ Just $ alterKey oldkey $ \d -> d
 		{ keyName = S.toShort $ encodeBS $ reSanitizeKeyName $ decodeBS $ S.fromShort $ keyName d }
 	| otherwise = return Nothing
diff --git a/Command/ExamineKey.hs b/Command/ExamineKey.hs
index b3a3cbab21..439472a47e 100644
--- a/Command/ExamineKey.hs
+++ b/Command/ExamineKey.hs
@@ -78,6 +78,6 @@ run o _ input = do
 		Just v -> getParsed v >>= \b ->
 			maybeLookupBackendVariety (fromKey keyVariety ik) >>= \case
 				Just ib -> case fastMigrate ib of
-					Just fm -> fromMaybe ik <$> fm ik b af
+					Just fm -> fromMaybe ik <$> fm ik b af False
 					Nothing -> pure ik
 				Nothing -> pure ik
diff --git a/Command/Migrate.hs b/Command/Migrate.hs
index c07377aae2..0e0dfbc67d 100644
--- a/Command/Migrate.hs
+++ b/Command/Migrate.hs
@@ -149,7 +149,7 @@ perform onlytweaksize o file oldkey oldkeyrec oldbackend newbackend = go =<< gen
 			}
 		newkey <- fst <$> genKey source nullMeterUpdate newbackend
 		return $ Just (newkey, False)
-	genkey (Just fm) = fm oldkey newbackend afile >>= \case
+	genkey (Just fm) = fm oldkey newbackend afile True >>= \case
 		Just newkey -> return (Just (newkey, True))
 		Nothing -> genkey Nothing
 	tweaksize k
diff --git a/Remote/Web.hs b/Remote/Web.hs
index 09efcc55e5..f953b8d929 100644
--- a/Remote/Web.hs
+++ b/Remote/Web.hs
@@ -12,7 +12,6 @@ import Types.Remote
 import Types.ProposedAccepted
 import Types.Creds
 import Types.Key
-import Types.KeySource
 import Remote.Helper.Special
 import Remote.Helper.ExportImport
 import qualified Git
@@ -31,7 +30,7 @@ import Annex.SpecialRemote.Config
 import Logs.Remote

(Diff truncated)
fix typo in example
diff --git a/doc/git-annex-examinekey.mdwn b/doc/git-annex-examinekey.mdwn
index b1f714ff56..bd8e19e596 100644
--- a/doc/git-annex-examinekey.mdwn
+++ b/doc/git-annex-examinekey.mdwn
@@ -62,7 +62,7 @@ that can be determined purely by looking at the key.
 
   Or to remove the extension from a key:
 
-  	git-annex examinekey SHA256E-xxx.tar.gz --migrate-to-backend=SHA256
+  	git-annex examinekey SHA256E--xxx.tar.gz --migrate-to-backend=SHA256
 
 * `--filename=name`
 

todo
diff --git a/doc/todo/speed_up_VURL_by_avoiding_redundant_hashing.mdwn b/doc/todo/speed_up_VURL_by_avoiding_redundant_hashing.mdwn
new file mode 100644
index 0000000000..56fdeced8b
--- /dev/null
+++ b/doc/todo/speed_up_VURL_by_avoiding_redundant_hashing.mdwn
@@ -0,0 +1,10 @@
+When a VURL key has multiple equivilant keys that all use the same hash,
+verifying the VURL key currently has to verify each equivilant key.
+Usually that is done incrementally, so it only has to read the file once. But it
+still does redundant work, updating each incremental verifier with each
+chunk.
+
+This could be improved by caclulating a hash once, and then compare it
+with a hash value exposed by the Backend. That seems doable but will mean
+extending the Backend interface, to expose the hash value and type.
+--[[Joey]] 

comment
diff --git a/doc/bugs/git-annex_is_slow_at_reading_file_content/comment_11_cab37d851237a7798f74867e08aaafb9._comment b/doc/bugs/git-annex_is_slow_at_reading_file_content/comment_11_cab37d851237a7798f74867e08aaafb9._comment
new file mode 100644
index 0000000000..5cffc2f2bf
--- /dev/null
+++ b/doc/bugs/git-annex_is_slow_at_reading_file_content/comment_11_cab37d851237a7798f74867e08aaafb9._comment
@@ -0,0 +1,33 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 11"""
+ date="2024-03-01T18:53:39Z"
+ content="""
+My laptop now has SHA256 in hardware I assume, as I'm seeing similar speed
+differences. Eg:
+
+	joey@darkstar:~/tmp/x3>time sha256sum x
+	e1f21b651362cf2f86ccc14a7376c33a49f30d7d14afe1ec6993eead5e8cfe41  x
+	0.53user 0.17system 0:00.71elapsed 99%CPU (0avgtext+0avgdata 3712maxresident)k
+	32inputs+0outputs (1major+239minor)pagefaults 0swaps
+	joey@darkstar:~/tmp/x3>time git-annex fsck x
+	fsck x (checksum...) ok
+	(recording state in git...)
+	3.22user 0.40system 0:03.63elapsed 100%CPU (0avgtext+0avgdata 61180maxresident)k
+	0inputs+144outputs (0major+8547minor)pagefaults 0swaps
+
+I agree that this bug should be left open since even a relatively low-end laptop
+now has this, git-annex shouldn't leave so much performance on the table.
+
+I've opened an issue on crypton:
+<https://github.com/kazu-yamamoto/crypton/issues/31>
+
+If it's rejected from crypton, I'm open to considering another library. 
+I'd rather avoid OpenSSL wrappers because adding a C library dependency on openssl
+will complicate building git-annex in some situations. It would be better to
+have a haskell library that, like cryptonite, embeds the necessary C code.
+
+Also, whatever library git-annex uses needs to support incremental hashing,
+otherwise git-annex has to pay a performance penalty of re-reading a file to hash
+it after download, rather than hashing while downloading.
+"""]]

update
diff --git a/CHANGELOG b/CHANGELOG
index cbb654d00c..2bc030c19d 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -3,7 +3,7 @@ git-annex (10.20240228) UNRELEASED; urgency=medium
   * addurl, importfeed: Added --verifiable option, which improves
     the safety of --fast or --relaxed by letting the content of
     annexed files be verified with a checksum that is calculated
-    on a later download from the web.
+    on a later download from the web. This will become the default later.
   * Added dependency on unbounded-delays.
 
  -- Joey Hess <id@joeyh.name>  Tue, 27 Feb 2024 13:07:10 -0400
diff --git a/doc/todo/migration_to_VURL_by_default.mdwn b/doc/todo/migration_to_VURL_by_default.mdwn
index 3d56464084..ce78eb857d 100644
--- a/doc/todo/migration_to_VURL_by_default.mdwn
+++ b/doc/todo/migration_to_VURL_by_default.mdwn
@@ -14,5 +14,10 @@ eventually. Could be a fsck warning about URL keys, at some point after
 transferring the content between repositories that it's not possible to
 verify it.
 
-Of course if users want to continue to use URL keys, that's fine. Users can
-also choose to use WORM keys if they really want to. --[[Joey]]
+Of course if users want to continue to use their existing URL keys and 
+not be able to verify content, that's fine. Users can also choose to use 
+WORM keys if they really want to. 
+
+However, I don't think there's enough reason to want to use URL keys to add
+configuration of which kind of keys addurl uses, once VURL is the default.
+--[[Joey]]

avoid double checksum when downloading VURL from web for 1st time
Sponsored-by: Jack Hill on Patreon
diff --git a/Remote/Web.hs b/Remote/Web.hs
index 40bcfc3494..09efcc55e5 100644
--- a/Remote/Web.hs
+++ b/Remote/Web.hs
@@ -147,27 +147,28 @@ downloadKey urlincludeexclude key _af dest p vc =
 			)
 
 	postdl v@Verified = return (Just v)
-	postdl v = do
-		when (fromKey keyVariety key == VURLKey) $
-			recordvurlkey
-		return (Just v)
+	postdl v
+		-- For a VURL key that was not verified on download, 
+		-- need to generate a hashed key for the content downloaded
+		-- from the web, and record it for later use verifying this
+		-- content.
+		--
+		-- But when the VURL key has a known size, and already has a
+		-- recorded hashed key, don't record a new key, since the
+		-- content on the web is expected to be stable for such a key.
+		| fromKey keyVariety key == VURLKey =
+			case fromKey keySize key of
+				Nothing -> 
+					getEquivilantKeys key
+						>>= recordvurlkey
+				Just _ -> do
+					eks <- getEquivilantKeys key
+					if null eks
+						then recordvurlkey eks
+						else return (Just v)
+		| otherwise = return (Just v)
 	
-	-- For a VURL key that was not verified on download, 
-	-- need to generate a hashed key for the content downloaded
-	-- from the web, and record it for later use verifying this content.
-	--
-	-- But when the VURL key has a known size, and already has a
-	-- recorded hashed key, don't record a new key, since the content
-	-- on the web is expected to be stable for such a key.
-	recordvurlkey = case fromKey keySize key of
-		Nothing -> recordvurlkey' =<< getEquivilantKeys key
-		Just _ -> do
-			eks <- getEquivilantKeys key
-			if null eks
-				then recordvurlkey' eks
-				else return ()
-	
-	recordvurlkey' eks = do
+	recordvurlkey eks = do
 		-- Make sure to pick a backend that is cryptographically
 		-- secure.
 		db <- defaultBackend
@@ -178,6 +179,7 @@ downloadKey urlincludeexclude key _af dest p vc =
 		(hashk, _) <- genKey ks nullMeterUpdate b
 		unless (hashk `elem` eks) $
 			setEquivilantKey key hashk
+		return (Just Verified)
 	  where
 		ks = KeySource
 			{ keyFilename = mempty -- avoid adding any extension
diff --git a/doc/todo/verified_relaxed_urls.mdwn b/doc/todo/verified_relaxed_urls.mdwn
index 6be4900eb3..30f83bff44 100644
--- a/doc/todo/verified_relaxed_urls.mdwn
+++ b/doc/todo/verified_relaxed_urls.mdwn
@@ -11,11 +11,7 @@ verify the content.
 The web special remote can hash the content as it's downloading it from the
 web, and record the resulting hash-based key.
 
-> Status: Working, but `git-annex addurl --relaxed --verifiable` followed
-> by `git-annex get` currently does 2 checksums in the get stage; it should
-> only do one.
-> 
-> It's not yet possible to migrate an URL key to a VURL key. Should be easy
+> Status: Working, but it's not yet possible to migrate an URL key to a VURL key. Should be easy
 > to add support for this. --[[Joey]]
 
 ## handling upgrades

incremental verification for VURL
Sponsored-by: Brett Eisenberg on Patreon
diff --git a/Backend/VURL.hs b/Backend/VURL.hs
index a02cc5848f..9a23967438 100644
--- a/Backend/VURL.hs
+++ b/Backend/VURL.hs
@@ -15,6 +15,8 @@ import Types.Key
 import Types.Backend
 import Logs.EquivilantKeys
 import Backend.Variety
+import Backend.Hash (descChecksum)
+import Utility.Hash
 
 backends :: [Backend]
 backends = [backendVURL]
@@ -30,14 +32,42 @@ backendVURL = Backend
 			-- because downloading the content from the web in
 			-- the first place records one.
 			[] -> return False
-			l -> do
+			eks -> do
 				let check ek = getbackend ek >>= \case
 					Nothing -> pure False
 					Just b -> case verifyKeyContent b of
 						Just verify -> verify ek f
 						Nothing -> pure False
-				anyM check l
-	, verifyKeyContentIncrementally = Nothing -- TODO
+				anyM check eks
+	, verifyKeyContentIncrementally = Just $ \k -> do
+		-- Run incremental verifiers for each equivilant key together,
+		-- and see if any of them succeed.
+		eks <- equivkeys k
+		let get = \ek -> getbackend ek >>= \case
+			Nothing -> pure Nothing
+			Just b -> case verifyKeyContentIncrementally b of
+				Nothing -> pure Nothing
+				Just va -> Just <$> va ek
+		l <- catMaybes <$> forM eks get
+		return $ IncrementalVerifier
+			{ updateIncrementalVerifier = \s ->
+				forM_ l $ flip updateIncrementalVerifier s
+			-- If there are no equivilant keys recorded somehow,
+			-- or if none of them support incremental verification,
+			-- this will return Nothing, which indicates that
+			-- incremental verification was not able to be
+			-- performed.
+			, finalizeIncrementalVerifier = do
+				r <- forM l finalizeIncrementalVerifier
+				return $ case catMaybes r of
+					[] -> Nothing
+					r' -> Just (or r')
+			, unableIncrementalVerifier = 
+				forM_ l unableIncrementalVerifier
+			, positionIncrementalVerifier =
+				getM positionIncrementalVerifier l
+			, descIncrementalVerifier = descChecksum
+			} 
 	, canUpgradeKey = Nothing
 	, fastMigrate = Nothing
 	-- Even if a hash is recorded on initial download from the web and
diff --git a/doc/todo/verified_relaxed_urls.mdwn b/doc/todo/verified_relaxed_urls.mdwn
index 751b7ea026..6be4900eb3 100644
--- a/doc/todo/verified_relaxed_urls.mdwn
+++ b/doc/todo/verified_relaxed_urls.mdwn
@@ -11,16 +11,12 @@ verify the content.
 The web special remote can hash the content as it's downloading it from the
 web, and record the resulting hash-based key.
 
-> Status: This is implemented and working, but verifyKeyContentIncrementally
-> needs to be implemented. Until it is, VURLs will not be as efficient 
-> as they could be.
+> Status: Working, but `git-annex addurl --relaxed --verifiable` followed
+> by `git-annex get` currently does 2 checksums in the get stage; it should
+> only do one.
 > 
-> Also `git-annex addurl --relaxed --verifiable` followed by `git-annex get`
-> currently does 2 checksums in the get stage; it should only do one.
-> Re-check this after implementing incremental verification.
-> 
-> It's not yet possible to migrate an URL key to a VURL key. Should
-> be easy to add support for this. --[[Joey]]
+> It's not yet possible to migrate an URL key to a VURL key. Should be easy
+> to add support for this. --[[Joey]]
 
 ## handling upgrades
 

has potential in DANDI project
diff --git a/doc/todo/importtree_only_remotes.mdwn b/doc/todo/importtree_only_remotes.mdwn
index f73a883b20..8f140c9450 100644
--- a/doc/todo/importtree_only_remotes.mdwn
+++ b/doc/todo/importtree_only_remotes.mdwn
@@ -76,4 +76,4 @@ Or by complicating Remote.Helper.ExportImport further..
 --[[Joey]]
 
 [[!tag confirmed]]
-[[!tag projects/datalad/potential]]
+[[!tag projects/dandi/potential]]

add future todo
diff --git a/doc/todo/migration_to_VURL_by_default.mdwn b/doc/todo/migration_to_VURL_by_default.mdwn
new file mode 100644
index 0000000000..3d56464084
--- /dev/null
+++ b/doc/todo/migration_to_VURL_by_default.mdwn
@@ -0,0 +1,18 @@
+`git-annex addurl --fast/--relaxed --verifiable` now uses VURL keys,
+which is an improvement over the old, un-verifiable URL keys. But users
+have to know to use it, and can have URL keys in their repository.
+
+Note that old git-annex, when in a repo with VURL keys, is still able to
+operate on them fine, even though it doesn't know what they are.
+Only fsck warns about them. So --verifiable could become the default
+reasonably soon. It's not necessary to wait for everyone to have the new
+version of git-annex.
+
+It might be good to nudge users to migrate their existing files to VURL
+eventually. Could be a fsck warning about URL keys, at some point after
+--verifiable becomes the default for addurl. Or could be a warning when
+transferring the content between repositories that it's not possible to
+verify it.
+
+Of course if users want to continue to use URL keys, that's fine. Users can
+also choose to use WORM keys if they really want to. --[[Joey]]
diff --git a/doc/todo/verified_relaxed_urls.mdwn b/doc/todo/verified_relaxed_urls.mdwn
index 3a313750e9..751b7ea026 100644
--- a/doc/todo/verified_relaxed_urls.mdwn
+++ b/doc/todo/verified_relaxed_urls.mdwn
@@ -42,6 +42,8 @@ That would leave it up to the user to migrate their URL keys to
 VURL keys, if desired. Now that distributed migration is
 implemented, that seems sufficiently easy.
 
+See [[migration_to_VURL_by_default]]
+
 ## addurl --fast 
 
 Using addurl --fast rather than --relaxed records the size but doesn't

verifyKeyContent for VURL
VURL is now fully working, though needs more testing.
Still need to implement verifyKeyContentIncrementally but it works
without it.
Sponsored-by: Luke T. Shumaker on Patreon
diff --git a/Backend/VURL.hs b/Backend/VURL.hs
index 002a7c061d..a02cc5848f 100644
--- a/Backend/VURL.hs
+++ b/Backend/VURL.hs
@@ -23,7 +23,20 @@ backendVURL :: Backend
 backendVURL = Backend
 	{ backendVariety = VURLKey
 	, genKey = Nothing
-	, verifyKeyContent = Nothing -- TODO
+	, verifyKeyContent = Just $ \k f -> do
+		equivkeys k >>= \case
+			-- Normally there will always be an key
+			-- recorded when a VURL's content is available,
+			-- because downloading the content from the web in
+			-- the first place records one.
+			[] -> return False
+			l -> do
+				let check ek = getbackend ek >>= \case
+					Nothing -> pure False
+					Just b -> case verifyKeyContent b of
+						Just verify -> verify ek f
+						Nothing -> pure False
+				anyM check l
 	, verifyKeyContentIncrementally = Nothing -- TODO
 	, canUpgradeKey = Nothing
 	, fastMigrate = Nothing
diff --git a/doc/todo/verified_relaxed_urls.mdwn b/doc/todo/verified_relaxed_urls.mdwn
index db1f857af9..3a313750e9 100644
--- a/doc/todo/verified_relaxed_urls.mdwn
+++ b/doc/todo/verified_relaxed_urls.mdwn
@@ -11,6 +11,17 @@ verify the content.
 The web special remote can hash the content as it's downloading it from the
 web, and record the resulting hash-based key.
 
+> Status: This is implemented and working, but verifyKeyContentIncrementally
+> needs to be implemented. Until it is, VURLs will not be as efficient 
+> as they could be.
+> 
+> Also `git-annex addurl --relaxed --verifiable` followed by `git-annex get`
+> currently does 2 checksums in the get stage; it should only do one.
+> Re-check this after implementing incremental verification.
+> 
+> It's not yet possible to migrate an URL key to a VURL key. Should
+> be easy to add support for this. --[[Joey]]
+
 ## handling upgrades
 
 A repository that currently contains the content of a relaxed url needs to
@@ -69,6 +80,12 @@ download on the fly). Then git-annex would take
 care of recording the hash-based key. The external special remote interface
 could be extended to include that.
 
+> For now, gonna punt on this. It would be possible to support other
+> special remotes later, but implemented it in the web special remote only
+> for now. When using `git-annex addurl --verified` with others, it creates
+> a VURL, but never generates a hash key, so the VURL works just like an
+> URL key.
+
 ## hash-based key choice
 
 Should annex.backend gitconfig be used to pick which hash-based key to use?
@@ -77,6 +94,11 @@ recorded for a VURL. Not really a problem, but would increase the
 size of the git-annex branch unncessarily, and require extra work when
 verifying the key.)
 
+> It will let annex.backend gitconfig and --backend be used,
+> but it didn't seem worth supporting annex.backend gitattribute, or really
+> even appropriate to since this is not really related to any particular
+> work tree file. --[[Joey]]
+
 What if annex.backend uses WORM or something that is not hash-based?
 Seems it ought to fall back to SHA256 or something then.
 
@@ -103,4 +125,11 @@ compared with migrating the key to the desired hash backend.
 Does seem to be some chance this could help implementing 
 [[wishlist_degraded_files]].
 
-[[!tag projects/datalad/potential]]
+## security
+
+There is the potential for a loop, where a VURL has recorded an equivilant
+key what is the same VURL, or another VURL in a loop. Leading to a crafted
+git-annex branch that DOSes git-annex.
+
+To avoid this, any VURL in equivilant keys will be ignored.
+

implement isCryptographicallySecureKey for VURL
Considerable difficulty to work around an import cycle. Had to move the
list of backends (except for VURL) to Backend.Variety to VURL could use
it.
Sponsored-by: Kevin Mueller on Patreon
diff --git a/Annex/Content.hs b/Annex/Content.hs
index 995eb6ed15..2800754216 100644
--- a/Annex/Content.hs
+++ b/Annex/Content.hs
@@ -439,7 +439,7 @@ moveAnnex key af src = ifM (checkSecureHashes' key)
 	alreadyhave = liftIO $ R.removeLink src
 
 checkSecureHashes :: Key -> Annex (Maybe String)
-checkSecureHashes key = ifM (Backend.isCryptographicallySecure key)
+checkSecureHashes key = ifM (Backend.isCryptographicallySecureKey key)
 	( return Nothing
 	, ifM (annexSecureHashesOnly <$> Annex.getGitConfig)
 		( return $ Just $ "annex.securehashesonly blocked adding " ++ decodeBS (formatKeyVariety (fromKey keyVariety key)) ++ " key"
diff --git a/Annex/Transfer.hs b/Annex/Transfer.hs
index 9eeecb7834..7bf0ca365b 100644
--- a/Annex/Transfer.hs
+++ b/Annex/Transfer.hs
@@ -42,7 +42,7 @@ import Types.WorkerPool
 import Annex.WorkerPool
 import Annex.TransferrerPool
 import Annex.StallDetection
-import Backend (isCryptographicallySecure)
+import Backend (isCryptographicallySecureKey)
 import Types.StallDetection
 import qualified Utility.RawFilePath as R
 
@@ -276,10 +276,10 @@ runTransferrer sd r k afile retrydecider direction _witness =
 preCheckSecureHashes :: Observable v => Key -> Maybe Backend -> Annex v -> Annex v
 preCheckSecureHashes k meventualbackend a = case meventualbackend of
 	Just eventualbackend -> go
-		(Types.Backend.isCryptographicallySecure eventualbackend)
+		(pure (Types.Backend.isCryptographicallySecure eventualbackend))
 		(Types.Backend.backendVariety eventualbackend)
 	Nothing -> go
-		(isCryptographicallySecure k)
+		(isCryptographicallySecureKey k)
 		(fromKey keyVariety k)
   where
 	go checksecure variety = ifM checksecure
diff --git a/Backend.hs b/Backend.hs
index a65a3cf4b6..10f2234a0c 100644
--- a/Backend.hs
+++ b/Backend.hs
@@ -18,10 +18,12 @@ module Backend (
 	lookupBuiltinBackendVariety,
 	maybeLookupBackendVariety,
 	isStableKey,
+	isCryptographicallySecureKey,
 	isCryptographicallySecure,
-	isCryptographicallySecure',
 ) where
 
+import qualified Data.Map as M
+
 import Annex.Common
 import qualified Annex
 import Annex.CheckAttr
@@ -29,18 +31,12 @@ import Types.Key
 import Types.KeySource
 import qualified Types.Backend as B
 import Utility.Metered
-
--- When adding a new backend, import it here and add it to the builtinList.
-import qualified Backend.Hash
-import qualified Backend.WORM
-import qualified Backend.URL
-import qualified Backend.External
-
-import qualified Data.Map as M
+import Backend.Variety
+import qualified Backend.VURL
 
 {- Built-in backends. Does not include externals. -}
 builtinList :: [Backend]
-builtinList = Backend.Hash.backends ++ Backend.WORM.backends ++ Backend.URL.backends
+builtinList = regularBackendList ++ Backend.VURL.backends
 
 {- The default hashing backend. This must use a cryptographically secure
  - hash. -}
@@ -107,25 +103,24 @@ lookupBuiltinBackendVariety :: KeyVariety -> Backend
 lookupBuiltinBackendVariety v = fromMaybe (giveup (unknownBackendVarietyMessage v)) $
 	maybeLookupBuiltinBackendVariety v
 
-maybeLookupBackendVariety :: KeyVariety -> Annex (Maybe Backend)
-maybeLookupBackendVariety (ExternalKey s hasext) =
-	Just <$> Backend.External.makeBackend s hasext
-maybeLookupBackendVariety v = 
-	pure $ M.lookup v varietyMap
-
 maybeLookupBuiltinBackendVariety :: KeyVariety -> Maybe Backend
 maybeLookupBuiltinBackendVariety v = M.lookup v varietyMap
 
+maybeLookupBackendVariety :: KeyVariety -> Annex (Maybe Backend)
+maybeLookupBackendVariety v = maybeLookupBackendVarietyMap v varietyMap
+
 varietyMap :: M.Map KeyVariety Backend
-varietyMap = M.fromList $ zip (map B.backendVariety builtinList) builtinList
+varietyMap = makeVarietyMap builtinList
 
 isStableKey :: Key -> Annex Bool
 isStableKey k = maybe False (`B.isStableKey` k) 
 	<$> maybeLookupBackendVariety (fromKey keyVariety k)
 
-isCryptographicallySecure :: Key -> Annex Bool
-isCryptographicallySecure k = maybe (pure False) isCryptographicallySecure'
+isCryptographicallySecureKey :: Key -> Annex Bool
+isCryptographicallySecureKey k = maybe 
+	(pure False)
+	(\b -> B.isCryptographicallySecureKey b k)
 	=<< maybeLookupBackendVariety (fromKey keyVariety k)
 
-isCryptographicallySecure' :: Backend -> Annex Bool
-isCryptographicallySecure' = B.isCryptographicallySecure
+isCryptographicallySecure :: Backend -> Bool
+isCryptographicallySecure = B.isCryptographicallySecure
diff --git a/Backend/External.hs b/Backend/External.hs
index 3feffd1ee1..b95cff5e3b 100644
--- a/Backend/External.hs
+++ b/Backend/External.hs
@@ -72,7 +72,8 @@ makeBackend' ebname@(ExternalBackendName bname) hasext (Right p) = do
 		, canUpgradeKey = Nothing
 		, fastMigrate = Nothing
 		, isStableKey = const isstable
-		, isCryptographicallySecure = pure iscryptographicallysecure
+		, isCryptographicallySecure = iscryptographicallysecure
+		, isCryptographicallySecureKey = const (pure iscryptographicallysecure)
 		}
 makeBackend' ebname hasext (Left _) = return $ unavailBackend ebname hasext
 
@@ -86,7 +87,8 @@ unavailBackend (ExternalBackendName bname) hasext =
 		, canUpgradeKey = Nothing
 		, fastMigrate = Nothing
 		, isStableKey = const False
-		, isCryptographicallySecure = pure False
+		, isCryptographicallySecure = False
+		, isCryptographicallySecureKey = const (pure False)
 		}
 
 genKeyExternal :: ExternalBackendName -> HasExt -> KeySource -> MeterUpdate -> Annex Key
diff --git a/Backend/Hash.hs b/Backend/Hash.hs
index e6a20bb7e0..3bf3bde1c5 100644
--- a/Backend/Hash.hs
+++ b/Backend/Hash.hs
@@ -81,7 +81,9 @@ genBackend hash = Backend
 	, canUpgradeKey = Just needsUpgrade
 	, fastMigrate = Just trivialMigrate
 	, isStableKey = const True
-	, isCryptographicallySecure = pure $ cryptographicallySecure hash
+	, isCryptographicallySecure = cryptographicallySecure hash
+	, isCryptographicallySecureKey = const $ pure $
+		cryptographicallySecure hash
 	}
 
 genBackendE :: Hash -> Backend
diff --git a/Backend/URL.hs b/Backend/URL.hs
index 0eeadaa289..af7427a104 100644
--- a/Backend/URL.hs
+++ b/Backend/URL.hs
@@ -1,5 +1,4 @@
-{- git-annex "URL" and "VURL" backends -- keys whose content is
- - available from urls.
+{- git-annex URL backend -- keys whose content is available from urls.
  -
  - Copyright 2011-2024 Joey Hess <id@joeyh.name>
  -
@@ -15,10 +14,9 @@ import Annex.Common
 import Types.Key
 import Types.Backend
 import Backend.Utilities
-import Logs.EquivilantKeys
 
 backends :: [Backend]
-backends = [backendURL, backendVURL]
+backends = [backendURL]
 
 backendURL :: Backend
 backendURL = Backend
@@ -31,25 +29,8 @@ backendURL = Backend
 	-- The content of an url can change at any time, so URL keys are
 	-- not stable.
 	, isStableKey = const False
-	, isCryptographicallySecure = pure False
-	}
-
-backendVURL :: Backend
-backendVURL = Backend
-	{ backendVariety = VURLKey
-	, genKey = Nothing
-	, verifyKeyContent = Nothing -- TODO
-	, verifyKeyContentIncrementally = Nothing -- TODO
-	, canUpgradeKey = Nothing
-	, fastMigrate = Nothing
-	-- Even if a hash is recorded on initial download from the web and
-	-- is used to verify every subsequent transfer including other
-	-- downloads from the web, in a split-brain situation there
-	-- can be more than one hash and different versions of the content.
-	-- So the content is not stable.
-	, isStableKey = const False
-	, isCryptographicallySecure = pure False 
-	-- TODO it is when all recorded hashes are

(Diff truncated)
add equivilant key log for VURL keys
When downloading a VURL from the web, make sure that the equivilant key
log is populated.
Unfortunately, this does not hash the content while it's being
downloaded from the web. There is not an interface in Backend currently
for incrementally hash generation, only for incremental verification of an
existing hash. So this might add a noticiable delay, and it has to show
a "(checksum...") message. This could stand to be improved.
But, that separate hashing step only has to happen on the first download
of new content from the web. Once the hash is known, the VURL key can have
its hash verified incrementally while downloading except when the
content in the web has changed. (Doesn't happen yet because
verifyKeyContentIncrementally is not implemented yet for VURL keys.)
Note that the equivilant key log file is formatted as a presence log.
This adds a tiny bit of overhead (eg "1 ") per line over just listing the
urls. The reason I chose to use that format is it seems possible that
there will need to be a way to remove an equivilant key at some point in
the future. I don't know why that would be necessary, but it seemed wise
to allow for the possibility.
Downloads of VURL keys from other special remotes that claim urls,
like bittorrent for example, does not popilate the equivilant key log.
So for now, no checksum verification will be done for those.
Sponsored-by: Nicholas Golder-Manning on Patreon
diff --git a/Backend.hs b/Backend.hs
index a1ea0184ad..0137993674 100644
--- a/Backend.hs
+++ b/Backend.hs
@@ -1,6 +1,6 @@
 {- git-annex key/value backends
  -
- - Copyright 2010-2021 Joey Hess <id@joeyh.name>
+ - Copyright 2010-2024 Joey Hess <id@joeyh.name>
  -
  - Licensed under the GNU AGPL version 3 or higher.
  -}
@@ -10,6 +10,7 @@
 module Backend (
 	builtinList,
 	defaultBackend,
+	defaultHashBackend,
 	genKey,
 	getBackend,
 	chooseBackend,
@@ -18,6 +19,7 @@ module Backend (
 	maybeLookupBackendVariety,
 	isStableKey,
 	isCryptographicallySecure,
+	isCryptographicallySecure',
 ) where
 
 import Annex.Common
@@ -40,7 +42,13 @@ import qualified Data.Map as M
 builtinList :: [Backend]
 builtinList = Backend.Hash.backends ++ Backend.WORM.backends ++ Backend.URL.backends
 
-{- Backend to use by default when generating a new key. -}
+{- The default hashing backend. This must use a cryptographically secure
+ - hash. -}
+defaultHashBackend :: Backend
+defaultHashBackend = Prelude.head builtinList
+
+{- Backend to use by default when generating a new key. Takes git config
+ - and --backend option into account. -}
 defaultBackend :: Annex Backend
 defaultBackend = maybe cache return =<< Annex.getState Annex.backend
   where
@@ -49,7 +57,7 @@ defaultBackend = maybe cache return =<< Annex.getState Annex.backend
 			=<< Annex.getRead Annex.forcebackend
 		b <- case n of
 			Just name | valid name -> lookupname name
-			_ -> pure (Prelude.head builtinList)
+			_ -> pure defaultHashBackend
 		Annex.changeState $ \s -> s { Annex.backend = Just b }
 		return b
 	valid name = not (null name)
@@ -116,5 +124,8 @@ isStableKey k = maybe False (`B.isStableKey` k)
 	<$> maybeLookupBackendVariety (fromKey keyVariety k)
 
 isCryptographicallySecure :: Key -> Annex Bool
-isCryptographicallySecure k = maybe False B.isCryptographicallySecure
+isCryptographicallySecure k = maybe False isCryptographicallySecure'
 	<$> maybeLookupBackendVariety (fromKey keyVariety k)
+
+isCryptographicallySecure' :: Backend -> Bool
+isCryptographicallySecure' = B.isCryptographicallySecure
diff --git a/Backend/Hash.hs b/Backend/Hash.hs
index 3fcba6a1fb..0c4ad61a0d 100644
--- a/Backend/Hash.hs
+++ b/Backend/Hash.hs
@@ -11,6 +11,7 @@ module Backend.Hash (
 	backends,
 	testKeyBackend,
 	keyHash,
+	descChecksum,
 ) where
 
 import Annex.Common
diff --git a/Backend/URL.hs b/Backend/URL.hs
index 32d3922396..209c6c843b 100644
--- a/Backend/URL.hs
+++ b/Backend/URL.hs
@@ -15,6 +15,7 @@ import Annex.Common
 import Types.Key
 import Types.Backend
 import Backend.Utilities
+import Logs.EquivilantKeys
 
 backends :: [Backend]
 backends = [backendURL, backendVURL]
diff --git a/Logs.hs b/Logs.hs
index d8d047cd9a..5e8daf5d0a 100644
--- a/Logs.hs
+++ b/Logs.hs
@@ -1,6 +1,6 @@
 {- git-annex log file names
  -
- - Copyright 2013-2023 Joey Hess <id@joeyh.name>
+ - Copyright 2013-2024 Joey Hess <id@joeyh.name>
  -
  - Licensed under the GNU AGPL version 3 or higher.
  -}
@@ -35,7 +35,9 @@ getLogVariety config f
 	| isRemoteStateLog f = Just NewUUIDBasedLog
 	| isRemoteContentIdentifierLog f = Just NewUUIDBasedLog
 	| isRemoteMetaDataLog f = Just RemoteMetaDataLog
-	| isMetaDataLog f || f `elem` otherTopLevelLogs = Just OtherLog
+	| isMetaDataLog f
+		|| f `elem` otherTopLevelLogs
+		|| isEquivilantKeyLog f = Just OtherLog
 	| otherwise = (LocationLog <$> locationLogFileKey config f)
 		<|> (ChunkLog <$> extLogFileKey chunkLogExt f)
 		<|> (UrlLog  <$> urlLogFileKey f)
@@ -70,6 +72,7 @@ keyLogFiles config k =
 	, remoteMetaDataLogFile config k
 	, remoteContentIdentifierLogFile config k
 	, chunkLogFile config k
+	, equivilantKeysLogFile config k
 	] ++ oldurlLogs config k
 
 {- All uuid-based logs stored in the top of the git-annex branch. -}
@@ -208,6 +211,18 @@ chunkLogFile config key =
 chunkLogExt :: S.ByteString
 chunkLogExt = ".log.cnk"
 
+{- The filename of the equivilant keys log for a given key. -}
+equivilantKeysLogFile :: GitConfig -> Key -> RawFilePath
+equivilantKeysLogFile config key = 
+	(branchHashDir config key P.</> keyFile key)
+		<> equivilantKeyLogExt
+
+equivilantKeyLogExt :: S.ByteString
+equivilantKeyLogExt = ".log.ek"
+
+isEquivilantKeyLog :: RawFilePath -> Bool
+isEquivilantKeyLog path = equivilantKeyLogExt `S.isSuffixOf` path
+
 {- The filename of the metadata log for a given key. -}
 metaDataLogFile :: GitConfig -> Key -> RawFilePath
 metaDataLogFile config key =
diff --git a/Logs/EquivilantKeys.hs b/Logs/EquivilantKeys.hs
new file mode 100644
index 0000000000..bf15f72364
--- /dev/null
+++ b/Logs/EquivilantKeys.hs
@@ -0,0 +1,31 @@
+{- Logs listing keys that are equivilant to a key.
+ -
+ - Copyright 2024 Joey Hess <id@joeyh.name>
+ -
+ - Licensed under the GNU AGPL version 3 or higher.
+ -}
+
+{-# LANGUAGE BangPatterns #-}
+
+module Logs.EquivilantKeys (
+	getEquivilantKeys,
+	setEquivilantKey,
+) where
+
+import Annex.Common
+import qualified Annex
+import Logs
+import Logs.Presence
+import qualified Annex.Branch
+
+getEquivilantKeys :: Key -> Annex [Key]
+getEquivilantKeys key = do
+	config <- Annex.getGitConfig
+	mapMaybe (deserializeKey' . fromLogInfo)
+		<$> presentLogInfo (equivilantKeysLogFile config key)
+
+setEquivilantKey :: Key -> Key -> Annex ()
+setEquivilantKey key equivkey = do
+	config <- Annex.getGitConfig
+	addLog (Annex.Branch.RegardingUUID []) (equivilantKeysLogFile config key)
+		InfoPresent (LogInfo (serializeKey' equivkey))
diff --git a/Remote/Web.hs b/Remote/Web.hs
index b3b788d2df..b1ab61dff2 100644
--- a/Remote/Web.hs
+++ b/Remote/Web.hs
@@ -1,6 +1,6 @@
 {- Web remote.
  -
- - Copyright 2011-2023 Joey Hess <id@joeyh.name>
+ - Copyright 2011-2024 Joey Hess <id@joeyh.name>
  -
  - Licensed under the GNU AGPL version 3 or higher.
  -}
@@ -11,6 +11,8 @@ import Annex.Common
 import Types.Remote
 import Types.ProposedAccepted
 import Types.Creds
+import Types.Key
+import Types.KeySource
 import Remote.Helper.Special
 import Remote.Helper.ExportImport
 import qualified Git
@@ -27,6 +29,9 @@ import qualified Annex.Url as Url
 import Annex.YoutubeDl
 import Annex.SpecialRemote.Config
 import Logs.Remote
+import Logs.EquivilantKeys
+import Backend

(Diff truncated)
support VURL backend
Not yet implemented is recording hashes on download from web and
verifying hashes.
addurl --verifiable option added with -V short option because I
expect a lot of people will want to use this.
It seems likely that --verifiable will become the default eventually,
and possibly rather soon. While old git-annex versions don't support
VURL, that doesn't prevent using them with keys that use VURL. Of
course, they won't verify the content on transfer, and fsck will warn
that it doesn't know about VURL. So there's not much problem with
starting to use VURL even when interoperating with old versions.
Sponsored-by: Joshua Antonishen on Patreon
diff --git a/Annex/Locations.hs b/Annex/Locations.hs
index 6e4f0faa2a..28494bed16 100644
--- a/Annex/Locations.hs
+++ b/Annex/Locations.hs
@@ -506,7 +506,7 @@ gitAnnexWebCertificate r = fromRawFilePath $ gitAnnexDir r P.</> "certificate.pe
 gitAnnexWebPrivKey :: Git.Repo -> FilePath
 gitAnnexWebPrivKey r = fromRawFilePath $ gitAnnexDir r P.</> "privkey.pem"
 
-{- .git/annex/feeds/ is used to record per-key (url) state by importfeeds -}
+{- .git/annex/feeds/ is used to record per-key (url) state by importfeed -}
 gitAnnexFeedStateDir :: Git.Repo -> RawFilePath
 gitAnnexFeedStateDir r = P.addTrailingPathSeparator $
 	gitAnnexDir r P.</> "feedstate"
diff --git a/Backend/URL.hs b/Backend/URL.hs
index 4df8979fb2..32d3922396 100644
--- a/Backend/URL.hs
+++ b/Backend/URL.hs
@@ -1,6 +1,7 @@
-{- git-annex "URL" backend -- keys whose content is available from urls.
+{- git-annex "URL" and "VURL" backends -- keys whose content is
+ - available from urls.
  -
- - Copyright 2011 Joey Hess <id@joeyh.name>
+ - Copyright 2011-2024 Joey Hess <id@joeyh.name>
  -
  - Licensed under the GNU AGPL version 3 or higher.
  -}
@@ -16,10 +17,10 @@ import Types.Backend
 import Backend.Utilities
 
 backends :: [Backend]
-backends = [backend]
+backends = [backendURL, backendVURL]
 
-backend :: Backend
-backend = Backend
+backendURL :: Backend
+backendURL = Backend
 	{ backendVariety = URLKey
 	, genKey = Nothing
 	, verifyKeyContent = Nothing
@@ -32,10 +33,28 @@ backend = Backend
 	, isCryptographicallySecure = False
 	}
 
+backendVURL :: Backend
+backendVURL = Backend
+	{ backendVariety = VURLKey
+	, genKey = Nothing
+	, verifyKeyContent = Nothing -- TODO
+	, verifyKeyContentIncrementally = Nothing -- TODO
+	, canUpgradeKey = Nothing
+	, fastMigrate = Nothing
+	-- Even if a hash is recorded on initial download from the web and
+	-- is used to verify every subsequent transfer including other
+	-- downloads from the web, in a split-brain situation there
+	-- can be more than one hash and different versions of the content.
+	-- So the content is not stable.
+	, isStableKey = const False
+	, isCryptographicallySecure = False 
+	-- TODO it is when all recorded hashes are
+	}
+
 {- Every unique url has a corresponding key. -}
-fromUrl :: String -> Maybe Integer -> Key
-fromUrl url size = mkKey $ \k -> k
+fromUrl :: String -> Maybe Integer -> Bool -> Key
+fromUrl url size verifiable = mkKey $ \k -> k
 	{ keyName = genKeyName url
-	, keyVariety = URLKey
+	, keyVariety = if verifiable then VURLKey else URLKey
 	, keySize = size
 	}
diff --git a/CHANGELOG b/CHANGELOG
index 2f454f0a19..cbb654d00c 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -1,5 +1,9 @@
 git-annex (10.20240228) UNRELEASED; urgency=medium
 
+  * addurl, importfeed: Added --verifiable option, which improves
+    the safety of --fast or --relaxed by letting the content of
+    annexed files be verified with a checksum that is calculated
+    on a later download from the web.
   * Added dependency on unbounded-delays.
 
  -- Joey Hess <id@joeyh.name>  Tue, 27 Feb 2024 13:07:10 -0400
diff --git a/Command/AddUrl.hs b/Command/AddUrl.hs
index d67456a05d..970089c02d 100644
--- a/Command/AddUrl.hs
+++ b/Command/AddUrl.hs
@@ -1,6 +1,6 @@
 {- git-annex command
  -
- - Copyright 2011-2021 Joey Hess <id@joeyh.name>
+ - Copyright 2011-2024 Joey Hess <id@joeyh.name>
  -
  - Licensed under the GNU AGPL version 3 or higher.
  -}
@@ -62,6 +62,7 @@ data AddUrlOptions = AddUrlOptions
 
 data DownloadOptions = DownloadOptions
 	{ relaxedOption :: Bool
+	, verifiableOption :: Bool
 	, rawOption :: Bool
 	, noRawOption :: Bool
 	, rawExceptOption :: Maybe (DeferredParse Remote)
@@ -96,7 +97,12 @@ parseDownloadOptions :: Bool -> Parser DownloadOptions
 parseDownloadOptions withfileoptions = DownloadOptions
 	<$> switch
 		( long "relaxed"
-		<> help "skip size check"
+		<> help "accept whatever content is downloaded from web even if it changes"
+		)
+	<*> switch
+		( long "verifiable"
+		<> short 'V'
+		<> help "improve later verification of --fast or --relaxed content"
 		)
 	<*> switch
 		( long "raw"
@@ -215,7 +221,7 @@ performRemote addunlockedmatcher r o uri file sz = lookupKey file >>= \case
 
 downloadRemoteFile :: AddUnlockedMatcher -> Remote -> DownloadOptions -> URLString -> RawFilePath -> Maybe Integer -> Annex (Maybe Key)
 downloadRemoteFile addunlockedmatcher r o uri file sz = checkCanAdd o file $ \canadd -> do
-	let urlkey = Backend.URL.fromUrl uri sz
+	let urlkey = Backend.URL.fromUrl uri sz (verifiableOption o)
 	createWorkTreeDirectory (parentDir file)
 	ifM (Annex.getRead Annex.fast <||> pure (relaxedOption o))
 		( do
@@ -344,7 +350,7 @@ downloadWeb :: AddUnlockedMatcher -> DownloadOptions -> URLString -> Url.UrlInfo
 downloadWeb addunlockedmatcher o url urlinfo file =
 	go =<< downloadWith' downloader urlkey webUUID url file
   where
-	urlkey = addSizeUrlKey urlinfo $ Backend.URL.fromUrl url Nothing
+	urlkey = addSizeUrlKey urlinfo $ Backend.URL.fromUrl url Nothing (verifiableOption o)
 	downloader f p = Url.withUrlOptions $ downloadUrl False urlkey p Nothing [url] f
 	go Nothing = return Nothing
 	go (Just (tmp, backend)) = ifM (useYoutubeDl o <&&> liftIO (isHtmlFile (fromRawFilePath tmp)))
@@ -388,7 +394,7 @@ downloadWeb addunlockedmatcher o url urlinfo file =
 							warning (UnquotedString dlcmd <> " did not download anything")
 							return Nothing
 		mediaurl = setDownloader url YoutubeDownloader
-		mediakey = Backend.URL.fromUrl mediaurl Nothing
+		mediakey = Backend.URL.fromUrl mediaurl Nothing (verifiableOption o)
 		-- Does the already annexed file have the mediaurl
 		-- as an url? If so nothing to do.
 		alreadyannexed dest k = do
@@ -436,7 +442,7 @@ startingAddUrl si url o p = starting "addurl" ai si $ do
 	-- used to prevent two threads running concurrently when that would
 	-- likely fail.
 	ai = OnlyActionOn urlkey (ActionItemOther (Just (UnquotedString url)))
-	urlkey = Backend.URL.fromUrl url Nothing
+	urlkey = Backend.URL.fromUrl url Nothing (verifiableOption (downloadOptions o))
 
 showDestinationFile :: RawFilePath -> Annex ()
 showDestinationFile file = do
@@ -539,12 +545,12 @@ nodownloadWeb addunlockedmatcher o url urlinfo file
 		return Nothing
   where
 	nomedia = do
-		let key = Backend.URL.fromUrl url (Url.urlSize urlinfo)
+		let key = Backend.URL.fromUrl url (Url.urlSize urlinfo) (verifiableOption o)
 		nodownloadWeb' o addunlockedmatcher url key file
 	usemedia mediafile = do
 		let dest = youtubeDlDestFile o file mediafile
 		let mediaurl = setDownloader url YoutubeDownloader
-		let mediakey = Backend.URL.fromUrl mediaurl Nothing
+		let mediakey = Backend.URL.fromUrl mediaurl Nothing (verifiableOption o)
 		nodownloadWeb' o addunlockedmatcher mediaurl mediakey dest
 
 youtubeDlDestFile :: DownloadOptions -> RawFilePath -> RawFilePath -> RawFilePath
diff --git a/Command/FromKey.hs b/Command/FromKey.hs
index 501ff41a99..292ab179a6 100644
--- a/Command/FromKey.hs
+++ b/Command/FromKey.hs
@@ -94,7 +94,7 @@ keyOpt = either giveup id . keyOpt'
 keyOpt' :: String -> Either String Key
 keyOpt' s = case parseURIPortable s of
 	Just u | not (isKeyPrefix (uriScheme u)) ->
-		Right $ Backend.URL.fromUrl s Nothing
+		Right $ Backend.URL.fromUrl s Nothing False
 	_ -> case deserializeKey s of
 		Just k -> Right k
 		Nothing -> Left $ "bad key/url " ++ s
diff --git a/Command/ImportFeed.hs b/Command/ImportFeed.hs
index f96bcd869c..eeb247980e 100644
--- a/Command/ImportFeed.hs
+++ b/Command/ImportFeed.hs
@@ -283,7 +283,7 @@ startDownload addunlockedmatcher opts cache cv todownload = case location todown
 	Enclosure url -> startdownloadenclosure url
 	MediaLink linkurl -> do
 		let mediaurl = setDownloader linkurl YoutubeDownloader
-		let mediakey = Backend.URL.fromUrl mediaurl Nothing
+		let mediakey = Backend.URL.fromUrl mediaurl Nothing (verifiableOption (downloadOptions opts))
 		-- Old versions of git-annex that used quvi might have
 		-- used the quviurl for this, so check if it's known
 		-- to avoid adding it a second time.
@@ -638,7 +638,7 @@ clearFeedProblem url =
 		=<< feedState url

(Diff truncated)
comment
diff --git a/doc/todo/add_--json-progress_support_in_push_and_pull/comment_4_0f3c22853b50550237f9659ed2120ab2._comment b/doc/todo/add_--json-progress_support_in_push_and_pull/comment_4_0f3c22853b50550237f9659ed2120ab2._comment
new file mode 100644
index 0000000000..24a52672ad
--- /dev/null
+++ b/doc/todo/add_--json-progress_support_in_push_and_pull/comment_4_0f3c22853b50550237f9659ed2120ab2._comment
@@ -0,0 +1,24 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 4"""
+ date="2024-02-27T17:22:52Z"
+ content="""
+Rather than using git-annex push/pull/sync with a complex json format,
+complications of knowing what remote it's acting on, etc, a program can
+simply use git-annex get/copy/drop/import/export to do the same operations,
+all of which already support json.
+
+> I'm OK with leaving Git operations silent for now.
+
+In the case where the git operation needs to prompt for a password, this
+would leave the user with a password prompt with no prior indication of what is
+being done. I don't think that's acceptable.
+
+> Please improve the end-user experience by devising a way to inform the
+> user git-annex is making progress with JSON.
+
+> I hope you find this information helpful to improve the end-user experience.
+> Let me know if you have any questions.
+
+Yikes, that almost triggered my ChatGPT detector.
+"""]]

fix link
diff --git a/doc/todo/add_--json-progress_support_in_push_and_pull/comment_1_403acf073a2a8d53eaa880cc2564347d._comment b/doc/todo/add_--json-progress_support_in_push_and_pull/comment_1_403acf073a2a8d53eaa880cc2564347d._comment
index 70de5b1d77..0f76888df8 100644
--- a/doc/todo/add_--json-progress_support_in_push_and_pull/comment_1_403acf073a2a8d53eaa880cc2564347d._comment
+++ b/doc/todo/add_--json-progress_support_in_push_and_pull/comment_1_403acf073a2a8d53eaa880cc2564347d._comment
@@ -5,7 +5,8 @@
  content="""
 They don't have --json, which would be a necessary first step.
 
-This was considered in [[this_todo|todo/--json_for_unannex__and_ideally_any_other_command_]]
+This was considered in
+[this_todo](https://git-annex.branchable.com/todo/--json_for_unannex__and_ideally_any_other_command_/)
 when adding --json to many commands, and the thinking for not adding it
 was:
 

add news item for git-annex 10.20240227
diff --git a/doc/news/version_10.20240227.mdwn b/doc/news/version_10.20240227.mdwn
new file mode 100644
index 0000000000..dfb95d996d
--- /dev/null
+++ b/doc/news/version_10.20240227.mdwn
@@ -0,0 +1,16 @@
+git-annex 10.20240227 released with [[!toggle text="these changes"]]
+[[!toggleable text="""  * importfeed: Added --scrape option, which uses yt-dlp to screen scrape
+    the equivilant of an RSS feed.
+  * importfeed --force: Don't treat it as a failure when an already
+    downloaded file exists. (Fixes a behavior change introduced in
+    10.20230626.)
+  * importfeed --force: Avoid creating duplicates of existing
+    already downloaded files when yt-dlp or a special remote was used.
+  * addurl, importfeed: Added --raw-except option.
+  * stack.yaml: Update to lts-22.9 and use crypton.
+  * assistant, undo: When committing, let the usual git commit
+    hooks run.
+  * Added annex.commitmessage-command config.
+  * pre-commit: Avoid committing the git-annex branch
+    (except when a commit is made in a view, which changes metadata).
+  * Pass --no-warnings to yt-dlp."""]]
\ No newline at end of file

comment and todo
diff --git a/doc/forum/git-annex_unused_on_directory_special_remote__63__/comment_1_5fb31042ed7c5c52969451b6fc0f4ddc._comment b/doc/forum/git-annex_unused_on_directory_special_remote__63__/comment_1_5fb31042ed7c5c52969451b6fc0f4ddc._comment
new file mode 100644
index 0000000000..9289a7bde6
--- /dev/null
+++ b/doc/forum/git-annex_unused_on_directory_special_remote__63__/comment_1_5fb31042ed7c5c52969451b6fc0f4ddc._comment
@@ -0,0 +1,12 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 1"""
+ date="2024-02-27T16:16:31Z"
+ content="""
+That file was created by a git-annex copy to the special remote that was
+interrupted before it finished. It's safe to remove it.
+
+`git-annex unused` can't detect these kinds of things on special remotes
+(though it can in git-annex repositories). I do think it would be good if
+it were able to. Opened [[todo/improve_unused_for_special_remotes]].
+"""]]
diff --git a/doc/todo/improve_unused_for_special_remotes.mdwn b/doc/todo/improve_unused_for_special_remotes.mdwn
new file mode 100644
index 0000000000..88bf5373ca
--- /dev/null
+++ b/doc/todo/improve_unused_for_special_remotes.mdwn
@@ -0,0 +1,29 @@
+A remote like the directory special remote can have objects that have not
+been fully transferred to it by an interrupted copy, that linger until the
+copy is re-run and the content gets fully sent to the remote. It would be
+good if `git-annex unused` could find and clean up such things, like it
+does for incomplete transfers into a git-annex repository.
+
+In the directory special remote, these are files named "tmp/$key/$key".
+
+This would need to be an extension to the remote interface to add an action
+to find when a key has such a file, and an action to delete one of them.
+
+A problem is that any such file might actually still be in the process
+of being sent, perhaps from a different repository than the one where
+`git-annex unused` is being run. So deleting such a file could cause that
+transfer to fail. This problem seems unavoidable generally.
+
+----
+
+It's also possible for a special remote to get keys stored in it which
+git-annex does not know about. For example, in a temporary clone of the
+git-annex repository, add a new file. Send it to the special remote. Then
+delete the temporary clone.
+
+`git-annex unused --from` can't detect those keys, because it can only ask
+the special remote about presence of keys that it knows about.
+
+Might it be possible to solve both problems together? Eg, add an action
+that has the special remote list all keys and partial keys present in it.
+--[[Joey]]

diff --git a/doc/forum/git-annex_unused_on_directory_special_remote__63__.mdwn b/doc/forum/git-annex_unused_on_directory_special_remote__63__.mdwn
new file mode 100644
index 0000000000..41956fa361
--- /dev/null
+++ b/doc/forum/git-annex_unused_on_directory_special_remote__63__.mdwn
@@ -0,0 +1,15 @@
+Should git-annex unused work on a directory special remote? I tried the following:
+
+	$ git-annex unused --from SanDisk64GB4Torrented
+	unused SanDisk64GB4Torrented (checking for unused data...) ok
+	$ 
+
+However, despite git-annex list --in SanDisk64GB4Torrented demonstrating that there were no annex files in it, the size of the directory was still large. And I can see a file in tmp/ of that annex's directory:
+
+	$ ls tmp/
+	SHA256E-s2351316992--bdfba5df0bd72cffdb398fe885d9e36d052617647c0ae4fd0579a8fc785c95ba.iso
+	$ 
+
+Am I right to think that the file being present in tmp/ should cause git-annex unused to list it?
+
+I'm assuming it's fine to just remove the file. But I guess this might be a bug... so I'm putting it out there just in case.

comment
diff --git a/doc/forum/rsync_test_remote_fail/comment_3_0b910e98485832dd7f305decdc39fa7f._comment b/doc/forum/rsync_test_remote_fail/comment_3_0b910e98485832dd7f305decdc39fa7f._comment
new file mode 100644
index 0000000000..b890f99d8a
--- /dev/null
+++ b/doc/forum/rsync_test_remote_fail/comment_3_0b910e98485832dd7f305decdc39fa7f._comment
@@ -0,0 +1,9 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 3"""
+ date="2024-02-26T19:40:15Z"
+ content="""
+Thanks for following up that it's fixed in current version.
+
+Please use [[bugs]] in the future for bug reports.
+"""]]

Added a comment: ignore, upgrading to v10.20231227 solved this issue
diff --git a/doc/forum/rsync_test_remote_fail/comment_2_824e45450cc01b30144ac5f8265b930e._comment b/doc/forum/rsync_test_remote_fail/comment_2_824e45450cc01b30144ac5f8265b930e._comment
new file mode 100644
index 0000000000..4eec6a6026
--- /dev/null
+++ b/doc/forum/rsync_test_remote_fail/comment_2_824e45450cc01b30144ac5f8265b930e._comment
@@ -0,0 +1,8 @@
+[[!comment format=mdwn
+ username="tchen_annex"
+ avatar="http://cdn.libravatar.org/avatar/758a07ad226fbf59a287e6cbff1c11ed"
+ subject="ignore, upgrading to v10.20231227 solved this issue"
+ date="2024-02-25T17:34:31Z"
+ content="""
+Okay, I tried the latest version of git-annex via git-annex-standalone package in NeuroDebian deb-src. Problem solved with latest version. Please this error report.
+"""]]

diff --git a/doc/forum/Backup_of_whole_Linux_system.mdwn b/doc/forum/Backup_of_whole_Linux_system.mdwn
index afafc55a08..1408c9a52b 100644
--- a/doc/forum/Backup_of_whole_Linux_system.mdwn
+++ b/doc/forum/Backup_of_whole_Linux_system.mdwn
@@ -1,4 +1,4 @@
-If I would want to backup my whole Linux system, what's unclear or maybe missing from Git Annex:
+If I would want to back up my whole Linux system, what's unclear or maybe missing from Git Annex:
 
 I'm not exactly sure about the best way to import the files. Should I just copy over all the files (e.g. using `cp -ax /* .`, or maybe `rsync -a /* .` or so) to the repo, and then use `git annex add`? (Let's skip `/dev` and maybe other special files for now.)
 

diff --git a/doc/contribute.mdwn b/doc/contribute.mdwn
index ed6b362da9..a005e22e81 100644
--- a/doc/contribute.mdwn
+++ b/doc/contribute.mdwn
@@ -2,7 +2,7 @@ Help make git-annex better!
 
 ## user support
 
-Hang out in git-annex support areas and answer user's questions.
+Hang out in git-annex support areas and answer users' questions.
 Don't be afraid to get an answer not 100% right, but avoid wild guesses.
 If you understand the basics of how git-annex works and know how to use it,
 you are ahead of many users, and can help them out tremendously.

Added a comment: this also fails with encryption=none
diff --git a/doc/forum/rsync_test_remote_fail/comment_1_deffffd5112a60ad7837479da045d39f._comment b/doc/forum/rsync_test_remote_fail/comment_1_deffffd5112a60ad7837479da045d39f._comment
new file mode 100644
index 0000000000..84bf746520
--- /dev/null
+++ b/doc/forum/rsync_test_remote_fail/comment_1_deffffd5112a60ad7837479da045d39f._comment
@@ -0,0 +1,8 @@
+[[!comment format=mdwn
+ username="tchen_annex"
+ avatar="http://cdn.libravatar.org/avatar/758a07ad226fbf59a287e6cbff1c11ed"
+ subject="this also fails with encryption=none"
+ date="2024-02-25T01:08:43Z"
+ content="""
+One potentially useful info. A bit more experimentation shows that this fails with encryption=none as well. Seems to be purely a rsync issue.
+"""]]

diff --git a/doc/forum/rsync_test_remote_fail.mdwn b/doc/forum/rsync_test_remote_fail.mdwn
new file mode 100644
index 0000000000..0e30d449c5
--- /dev/null
+++ b/doc/forum/rsync_test_remote_fail.mdwn
@@ -0,0 +1,84 @@
+Hi,
+
+I am running into a bizarre testremote failure (with an rsync+encryption=shared special remote). 
+
+Here are the exact steps to replicate:
+
+    $ mkdir test
+    $ cd test/
+    $ git init
+    $ git annex init
+    $ git annex initremote newremote type=rsync rsyncurl=10.0.0.100:/backup/annex-enc/test encryption=shared
+    $ git annex testremote newremote
+
+    Here is what I will get
+
+    testremote newremote (generating test keys...) Remote Tests
+      unavailable remote
+        removeKey:                                       OK
+        storeKey:                                        OK
+        checkPresent:                                    OK
+        retrieveKeyFile:                                 OK
+        retrieveKeyFileCheap:                            OK
+      key size Just 1048576; remote chunksize=0 encryption=none
+        removeKey when not present:                      OK (1.51s)
+        present False:                                   OK (0.62s)
+        storeKey:                                        OK (1.10s)
+        present True:                                    FAIL (0.62s)
+          ./Command/TestRemote.hs:290:
+          failed
+        storeKey when already present:                   OK (0.37s)
+        present True:                                    FAIL (0.63s)
+          ./Command/TestRemote.hs:290:
+          failed
+        retrieveKeyFile:                                 rsync: change_dir "/backup/annex-enc/test/242/793/'SHA256E-s1048576--26a83e410604bf3f0f9288c14b39387ce244a832a02b192c81a97a36a0e42296.this-is-a-test-key" failed: No such file or directory (2)
+    rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1865) [Receiver=3.2.7]
+    rsync: [Receiver] write error: Broken pipe (32)
+    rsync exited 23
+    rsync: change_dir "/backup/annex-enc/test/W4/4g/'SHA256E-s1048576--26a83e410604bf3f0f9288c14b39387ce244a832a02b192c81a97a36a0e42296.this-is-a-test-key" failed: No such file or directory (2)
+    rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1865) [Receiver=3.2.7]
+    rsync: [Receiver] write error: Broken pipe (32)
+    rsync exited 23
+    FAIL (0.61s)
+          ./Command/TestRemote.hs:290:
+          failed
+        fsck downloaded object:                          OK
+        retrieveKeyFile resume from 33%:                 FAIL
+          Exception: .git/annex/objects/W4/4g/SHA256E-s1048576--26a83e410604bf3f0f9288c14b39387ce244a832a02b192c81a97a36a0e42296.this-is-a-test-key/SHA256E-s1048576--26a83e410604bf3f0f9288c14b39387ce244a832a02b192c81a97a36a0e42296.this-is-a-test-key: openBinaryFile: does not exist (No such file or directory)
+        fsck downloaded object:                          OK
+        retrieveKeyFile resume from 0:                   rsync: change_dir "/backup/annex-enc/test/242/793/'SHA256E-s1048576--26a83e410604bf3f0f9288c14b39387ce244a832a02b192c81a97a36a0e42296.this-is-a-test-key" failed: No such file or directory (2)
+    rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1865) [Receiver=3.2.7]
+    rsync: [Receiver] write error: Broken pipe (32)
+    rsync exited 23
+    rsync: change_dir "/backup/annex-enc/test/W4/4g/'SHA256E-s1048576--26a83e410604bf3f0f9288c14b39387ce244a832a02b192c81a97a36a0e42296.this-is-a-test-key" failed: No such file or directory (2)
+    rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1865) [Receiver=3.2.7]
+    rsync: [Receiver] write error: Broken pipe (32)
+    rsync exited 23
+    FAIL (0.61s)
+          ./Command/TestRemote.hs:290:
+          failed
+        fsck downloaded object:                          OK
+        retrieveKeyFile resume from end:                 cp: cannot stat '.git/annex/objects/W4/4g/SHA256E-s1048576--26a83e410604bf3f0f9288c14b39387ce244a832a02b192c81a97a36a0e42296.this-is-a-test-key/SHA256E-s1048576--26a83e410604bf3f0f9288c14b39387ce244a832a02b192c81a97a36a0e42296.this-is-a-test-key': No such file or directory
+
+This will repeat again and again (with different remote chunksize?) until finally it hangs where I would have to Ctrl+C it to force it to abort.
+
+I am using Ubuntu 22.04.3 LTS (Jammy) with the following git annex version:
+
+    $ git annex version
+    git-annex version: 8.20210223
+    build flags: Assistant Webapp Pairing Inotify DBus DesktopNotify TorrentParser MagicMime Feeds Testsuite S3 WebDAV
+    dependency versions: aws-0.22 bloomfilter-2.0.1.0 cryptonite-0.26 DAV-1.3.4 feed-1.3.0.1 ghc-8.8.4 http-client-0.6.4.1 persistent-sqlite-2.10.6.2 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.1.0
+    key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL X*
+    remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs httpalso borg hook external
+    operating system: linux x86_64
+    supported repository versions: 8
+    upgrade supported from repository versions: 0 1 2 3 4 5 6 7
+    local repository version: 8
+
+I can't see why git-annex rsync would be failing. I can do normal rsync via command line w/o any issues.
+
+Any help would be appreciated.
+
+
+
+

comment
diff --git a/doc/submodules/comment_11_78fffae9facce1600915bc19429d81e2._comment b/doc/submodules/comment_11_78fffae9facce1600915bc19429d81e2._comment
new file mode 100644
index 0000000000..8e7289cc69
--- /dev/null
+++ b/doc/submodules/comment_11_78fffae9facce1600915bc19429d81e2._comment
@@ -0,0 +1,17 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""Re: comment 6"""
+ date="2024-02-22T16:50:34Z"
+ content="""
+@TTTTAAAx kindly posted a full example of their problem, 
+which I've moved to 
+[[todo/detect_and_handle_submodules_after_path_changed_by_mv]].
+
+I do think that using `git mv` to rename directories that contain
+submodules is the right way to avoid that kind of problem. 
+Note that renaming such a directory without using git followed by running
+`git add` on the new directory has the same behavior as running
+`git-annex assist` does. This is not a git-annex problem, but I think it
+could be considered a git problem; git could make `git add` of a moved
+submodule do the right thing.
+"""]]

close
diff --git a/doc/todo/detect_and_handle_submodules_after_path_changed_by_mv.mdwn b/doc/todo/detect_and_handle_submodules_after_path_changed_by_mv.mdwn
index d954204ad4..2e41a31131 100644
--- a/doc/todo/detect_and_handle_submodules_after_path_changed_by_mv.mdwn
+++ b/doc/todo/detect_and_handle_submodules_after_path_changed_by_mv.mdwn
@@ -70,4 +70,6 @@ git annex assist
 echo
 
 echo test done
-```
+``
+
+> [[wontfix|done]] as it is out of scope --[[Joey]]`

comment
diff --git a/doc/todo/detect_and_handle_submodules_after_path_changed_by_mv/comment_1_9f9da126a031d2a4c98555b381dbcbdc._comment b/doc/todo/detect_and_handle_submodules_after_path_changed_by_mv/comment_1_9f9da126a031d2a4c98555b381dbcbdc._comment
new file mode 100644
index 0000000000..1c315954de
--- /dev/null
+++ b/doc/todo/detect_and_handle_submodules_after_path_changed_by_mv/comment_1_9f9da126a031d2a4c98555b381dbcbdc._comment
@@ -0,0 +1,11 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 1"""
+ date="2024-02-22T16:43:01Z"
+ content="""
+AFAICS, the behavior is the same if you do not use git-annex at all and mv
+submodules around generally.
+
+This strongly suggests that dealing with this is out of scope for
+git-annex.
+"""]]

move misplaced bug or todo to a better place
diff --git a/doc/submodules/bug.mdwn b/doc/todo/detect_and_handle_submodules_after_path_changed_by_mv.mdwn
similarity index 100%
rename from doc/submodules/bug.mdwn
rename to doc/todo/detect_and_handle_submodules_after_path_changed_by_mv.mdwn

Added a comment: Datalad cannot detect submodule path changed on disk
diff --git a/doc/submodules/comment_10_936e145bbe3b6bd928571c17d3416937._comment b/doc/submodules/comment_10_936e145bbe3b6bd928571c17d3416937._comment
new file mode 100644
index 0000000000..eda12b3fb8
--- /dev/null
+++ b/doc/submodules/comment_10_936e145bbe3b6bd928571c17d3416937._comment
@@ -0,0 +1,12 @@
+[[!comment format=mdwn
+ username="TTTTAAAx"
+ avatar="http://cdn.libravatar.org/avatar/9edd4b69b9f9fc9b8c1cb8ecd03902d5"
+ subject="Datalad cannot detect submodule path changed on disk"
+ date="2024-02-22T08:34:47Z"
+ content="""
+> Sounds like you might want to use datalad, which is built around git annex and where submodules are a first-class citizen. 
+
+Datalad handles submodules as subdatasets and add python code layers on it to handle datasets(e.g. dedup submodules). But it doesn't detect the submodules path changed like git.
+
+So, it doesn't do my needs sadly.
+"""]]

http://git-annex.branchable.com/submodules/#comment-2f017a0f1a13329f4f738568b3eb82ae
diff --git a/doc/submodules/bug.mdwn b/doc/submodules/bug.mdwn
new file mode 100644
index 0000000000..d954204ad4
--- /dev/null
+++ b/doc/submodules/bug.mdwn
@@ -0,0 +1,73 @@
+It's an enhancement feature to handle submodules to manage data with associated its projects.
+
+I want `git-annex` could detect submodule paths changed on disks which was cause by `mv` or file explorer.
+If user uses `git-annex-assist daemon` or `git-annex-assist` command directly after `mv` command, The submodules would be totally broken.
+
+Currently, the workaround is just use `git-mv` on each submodules manually.
+
+I made a testing shell script for this. 
+
+
+```shell
+#!/bin/bash
+# This is test script for submodule path changing.
+# set -e
+USE_GIT_MV=false # USE_GIT_MV=true works correctly
+cd /tmp/
+mkdir -p test_sub/{archive/projects,projects/2023_01_personal_some_cool_project,resources}
+cd test_sub
+git init
+git annex init
+git annex version
+cd projects/2023_01_personal_some_cool_project
+
+echo NOTE: Add some data and sub-projects for testing
+touch README.md 01_dataset_lists.csv 09_reports.md
+git submodule add https://github.com/Lykos153/git-annex-remote-googledrive.git
+git submodule add https://github.com/alpernebbi/git-annex-adapter.git
+git submodule status # check it
+git annex assist
+echo
+
+echo NOTE: I think that the projects are need to be changed "01_Projects" for sorting order.
+cd /tmp/test_sub
+if $USE_GIT_MV; then
+    git mv projects 01_Projects
+else
+    # NOTE: Just rename file makes submodules broken. directory depth is same
+    mv projects 01_Projects
+    (
+        cd 01_Projects/2023_01_personal_some_cool_project/git-annex-adapter
+        git status # it shows 'No such file or directory'
+    )
+fi
+git submodule status # check it
+git annex assist
+echo
+
+echo NOTE: I want to change some submodule name is for referencing just for work.
+cd /tmp/test_sub/01_Projects/2023_01_personal_some_cool_project
+if $USE_GIT_MV; then
+    git mv git-annex-adapter ref_sample_adapter_code
+else
+    # NOTE: Just rename file makes submodules broken. directory depth is same
+    mv git-annex-adapter ref_sample_adapter_code
+fi
+git submodule status # check it
+git annex assist
+echo
+
+echo NOTE: Now, i want to archive my old projects.
+cd /tmp/test_sub
+if $USE_GIT_MV; then
+    git mv 01_Projects/2023_01_personal_some_cool_project archive/projects/2023_01_personal_some_cool_project
+else
+    # NOTE: Just rename file makes submodules broken. directory depth is changed
+    mv 01_Projects/2023_01_personal_some_cool_project archive/projects/2023_01_personal_some_cool_project
+fi
+git submodule status # check it
+git annex assist
+echo
+
+echo test done
+```

Added a comment
diff --git a/doc/forum/copying_annex_between_remotes_manually__63__/comment_2_1237297cfb596d59a48c8359d43a3928._comment b/doc/forum/copying_annex_between_remotes_manually__63__/comment_2_1237297cfb596d59a48c8359d43a3928._comment
new file mode 100644
index 0000000000..b1abc59cc4
--- /dev/null
+++ b/doc/forum/copying_annex_between_remotes_manually__63__/comment_2_1237297cfb596d59a48c8359d43a3928._comment
@@ -0,0 +1,8 @@
+[[!comment format=mdwn
+ username="lell"
+ avatar="http://cdn.libravatar.org/avatar/4c4138a71d069e290240a3a12367fabe"
+ subject="comment 2"
+ date="2024-02-22T07:49:58Z"
+ content="""
+In the example above, of course I forgot one line that creates the file \"file\"  in the repository gta, e.g. like this `echo test > file`
+"""]]

Added a comment: Using fsck is an option?
diff --git a/doc/forum/copying_annex_between_remotes_manually__63__/comment_1_c13a0cded3a5fee06106f101f122eb33._comment b/doc/forum/copying_annex_between_remotes_manually__63__/comment_1_c13a0cded3a5fee06106f101f122eb33._comment
new file mode 100644
index 0000000000..ca7ba701a9
--- /dev/null
+++ b/doc/forum/copying_annex_between_remotes_manually__63__/comment_1_c13a0cded3a5fee06106f101f122eb33._comment
@@ -0,0 +1,50 @@
+[[!comment format=mdwn
+ username="lell"
+ avatar="http://cdn.libravatar.org/avatar/4c4138a71d069e290240a3a12367fabe"
+ subject="Using fsck is an option?"
+ date="2024-02-22T07:47:36Z"
+ content="""
+Hi, this worked for me:
+
+```
+~$ mkdir ~/test/gta
+~$ cd ~/test/gta
+~/test/gta$ git init
+
+Initialized empty Git repository in /home/lell/test/gta/.git/
+
+~/test/gta$ git annex init
+~/test/gta$ git annex add file
+~/test/gta$ git commit -m test
+
+~/test/gta$ cd ..
+~/test$ git clone a b
+
+~/test$ cd gta/
+~/test/gta$ cd ..
+~/test$ git clone gta gta2
+~/test$ cd gta2
+
+~/test/gta2$ mkdir .git/annex/objects
+~/test/gta2$ cp -r ../gta/.git/annex/objects/* .git/annex/objects/
+~/test/gta2$ cp -r ../gta/.git/annex/objects .git/annex/objects
+~/test/gta2$ git annex list   # it does not yet know that the file is now also here
+here
+|origin
+||web
+|||bittorrent
+||||
+_X__ file
+~/test/gta2$ git annex fsck
+fsck file (fixing location log) (checksum...) ok
+(recording state in git...)
+$ git annex list   # now it knows
+here
+|origin
+||web
+|||bittorrent
+||||
+XX__ file
+```
+
+"""]]

Added a comment: Colocating git-annex and git-lfs
diff --git a/doc/tips/storing_data_in_git-lfs/comment_1_e4eb406a4bf146c1f442047a4c1b1e1a._comment b/doc/tips/storing_data_in_git-lfs/comment_1_e4eb406a4bf146c1f442047a4c1b1e1a._comment
new file mode 100644
index 0000000000..24908e6a6a
--- /dev/null
+++ b/doc/tips/storing_data_in_git-lfs/comment_1_e4eb406a4bf146c1f442047a4c1b1e1a._comment
@@ -0,0 +1,24 @@
+[[!comment format=mdwn
+ username="beryllium@5bc3c32eb8156390f96e363e4ba38976567425ec"
+ nickname="beryllium"
+ avatar="http://cdn.libravatar.org/avatar/62b67d68e918b381e7e9dd6a96c16137"
+ subject="Colocating git-annex and git-lfs"
+ date="2024-02-22T02:55:25Z"
+ content="""
+Is it possible to add git-lfs capabilities to a git-annex, without using a special remote?
+
+I guess what I want is, are there any reasonable instructions to graft the hooks so that this is possible:
+
+
+	$ git init
+	$ git-lfs install
+	$ git-annex init
+
+And you can alternate between something like below:
+
+	$ git-lfs track \"*.exif_thumbnail.*\"
+	$ git-annex add IMG_0001.jpg
+	$ git add IMG_0001.exif_thumbnail.jpg
+
+Obviously this betrays the scenario of extracting thumbnails from the EXIF header and storing them alongside, as another form of metadata. If there's a better workflow to this, that would be appreciated too.
+"""]]

Added a comment: Multi-line string in WHEREIS-SUCCESS?
diff --git a/doc/design/external_special_remote_protocol/comment_53_a2d1778e6d61f9cb606fb83f0a4a73f4._comment b/doc/design/external_special_remote_protocol/comment_53_a2d1778e6d61f9cb606fb83f0a4a73f4._comment
new file mode 100644
index 0000000000..5669cae726
--- /dev/null
+++ b/doc/design/external_special_remote_protocol/comment_53_a2d1778e6d61f9cb606fb83f0a4a73f4._comment
@@ -0,0 +1,10 @@
+[[!comment format=mdwn
+ username="matrss"
+ avatar="http://cdn.libravatar.org/avatar/59541f50d845e5f81aff06e88a38b9de"
+ subject="Multi-line string in WHEREIS-SUCCESS?"
+ date="2024-02-21T12:29:14Z"
+ content="""
+Is it possible to somehow make `git annex whereis` show the response of the special remote to `WHEREIS` over multiple lines? Just including newlines obviously results in an error, since that ends the WHEREIS-SUCCESS message.
+
+I am implementing a special remote for which the data is fully described by what is essentially a json-encoded request to a third-party API, and I would like to show this json string pretty-printed over multiple lines in the whereis output, instead of as a single line.
+"""]]

diff --git a/doc/forum/copying_annex_between_remotes_manually__63__.mdwn b/doc/forum/copying_annex_between_remotes_manually__63__.mdwn
new file mode 100644
index 0000000000..6e3fc6d7a6
--- /dev/null
+++ b/doc/forum/copying_annex_between_remotes_manually__63__.mdwn
@@ -0,0 +1,5 @@
+I'm currently trying to migrate a git-annex repository to a new machine, one of whose remotes is the one from [this issue](https://git-annex.branchable.com/bugs/apparent_hang_in_git-annex-smudge/). I determined that the root cause there seemed to be under git rather than git-annex; in particular, any whole-repository operation would take multiple days to execute, for unclear reasons. Pulling the commit data to a new repository seems to fix this.
+
+I'm now trying to move all the annexed data from the original, broken remote to the new one. The default option here would be `git annex move`. However, when I run this, it apparently does some git operation for every key moved, taking hours to days; there are tens of thousands of keys, so this is obviously unworkable.
+
+Is there a way to simply mass-move the annexed data into the new repo, via rsync or similar, and then update the new repo's metadata all at once? The state of the old repository does not matter, since I intend to discard it as soon as this migration is done.

Add logo version with '> git annex' text below
diff --git a/doc/logo-with-cli.png b/doc/logo-with-cli.png
new file mode 100644
index 0000000000..40e47fdb5d
Binary files /dev/null and b/doc/logo-with-cli.png differ
diff --git a/doc/logo-with-cli.svg b/doc/logo-with-cli.svg
new file mode 100644
index 0000000000..9d30581484
--- /dev/null
+++ b/doc/logo-with-cli.svg
@@ -0,0 +1,107 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<!-- Created with Inkscape (http://www.inkscape.org/) -->
+
+<svg
+   id="svg2"
+   sodipodi:version="0.32"
+   inkscape:version="1.3.2 (091e20ef0f, 2023-11-25, custom)"
+   width="337.62668"
+   height="275.14575"
+   version="1.0"
+   sodipodi:docname="logo-with-cli.svg"
+   xml:space="preserve"
+   inkscape:export-filename="logo-with-cli.png"
+   inkscape:export-xdpi="300"
+   inkscape:export-ydpi="300"
+   xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
+   xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
+   xmlns="http://www.w3.org/2000/svg"
+   xmlns:svg="http://www.w3.org/2000/svg"
+   xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
+   xmlns:cc="http://creativecommons.org/ns#"
+   xmlns:dc="http://purl.org/dc/elements/1.1/"><metadata
+     id="metadata7"><rdf:RDF><cc:Work
+         rdf:about="Git Logo"><dc:format>image/svg+xml</dc:format><dc:type
+           rdf:resource="http://purl.org/dc/dcmitype/StillImage" /></cc:Work></rdf:RDF></metadata><defs
+     id="defs5"><inkscape:perspective
+       sodipodi:type="inkscape:persp3d"
+       inkscape:vp_x="0 : 101.13537 : 1"
+       inkscape:vp_y="0 : 1000 : 0"
+       inkscape:vp_z="97 : 101.13537 : 1"
+       inkscape:persp3d-origin="48.5 : 69.802067 : 1"
+       id="perspective11" /><inkscape:perspective
+       id="perspective3682"
+       inkscape:persp3d-origin="0.5 : 7.4687698 : 1"
+       inkscape:vp_z="1 : 7.6353698 : 1"
+       inkscape:vp_y="0 : 1000 : 0"
+       inkscape:vp_x="0 : 7.6353698 : 1"
+       sodipodi:type="inkscape:persp3d" /><inkscape:perspective
+       sodipodi:type="inkscape:persp3d"
+       inkscape:vp_x="0 : -140.76773 : 1"
+       inkscape:vp_y="0 : 1000 : 0"
+       inkscape:vp_z="97 : -140.76773 : 1"
+       inkscape:persp3d-origin="48.5 : -172.10103 : 1"
+       id="perspective11-7" /><inkscape:perspective
+       id="perspective3682-8"
+       inkscape:persp3d-origin="0.5 : -234.43433 : 1"
+       inkscape:vp_z="1 : -234.26773 : 1"
+       inkscape:vp_y="0 : 1000 : 0"
+       inkscape:vp_x="0 : -234.26773 : 1"
+       sodipodi:type="inkscape:persp3d" /></defs><sodipodi:namedview
+     inkscape:window-height="1517"
+     inkscape:window-width="2560"
+     inkscape:pageshadow="2"
+     inkscape:pageopacity="0.0"
+     guidetolerance="10.0"
+     gridtolerance="10.0"
+     objecttolerance="10.0"
+     borderopacity="1.0"
+     bordercolor="#666666"
+     pagecolor="#ffffff"
+     id="base"
+     inkscape:zoom="1.6910175"
+     inkscape:cx="20.401918"
+     inkscape:cy="180.36478"
+     inkscape:window-x="0"
+     inkscape:window-y="0"
+     inkscape:current-layer="svg2"
+     showguides="false"
+     showgrid="false"
+     inkscape:window-maximized="1"
+     inkscape:showpageshadow="2"
+     inkscape:pagecheckerboard="0"
+     inkscape:deskcolor="#d1d1d1"
+     showborder="true" /><text
+     xml:space="preserve"
+     style="font-size:48.1739px;line-height:1.25;font-family:Frutiger;-inkscape-font-specification:'Frutiger, Normal';font-variation-settings:normal;word-spacing:0px;vector-effect:none;fill:#666666;fill-opacity:1;stroke-width:0.735607;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1;-inkscape-stroke:none;stop-color:#000000"
+     x="18.510771"
+     y="248.40559"
+     id="text1-7"><tspan
+       sodipodi:role="line"
+       id="tspan1-0"
+       x="18.510771"
+       y="248.40559"
+       style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:48.1739px;font-family:'JetBrains Mono';-inkscape-font-specification:'JetBrains Mono, Bold';font-variant-ligatures:normal;font-variant-caps:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-variation-settings:normal;vector-effect:none;fill:#666666;fill-opacity:1;stroke-width:0.735607;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1;-inkscape-stroke:none;stop-color:#000000;stop-opacity:1"
+       dx="0 0 0"><tspan
+   style="font-variation-settings:normal;letter-spacing:0px;vector-effect:none;fill:#666666;fill-opacity:1;stroke-width:0.735607;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1;-inkscape-stroke:none;stop-color:#000000;stop-opacity:1"
+   id="tspan3-4"><tspan
+   style="font-variation-settings:normal;vector-effect:none;fill:#666666;fill-opacity:1;stroke-width:0.735607;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1;-inkscape-stroke:none;stop-color:#000000;stop-opacity:1"
+   id="tspan4-0">&gt;</tspan> </tspan>git annex</tspan></text><path
+     sodipodi:nodetypes="ccccccccccccccccccc"
+     id="path1927-4-1"
+     d="m 160.79525,192.35577 c -11.60008,-1.49439 -23.53847,-12.46256 -25.20639,-25.33023 -1.59972,-10.75758 0.70817,-23.31443 9.6007,-30.41399 3.90042,-2.65848 9.20606,-5.63712 13.88482,-5.28265 0.14013,4.76254 -0.14013,10.55125 0,15.31379 -5.57282,0.62044 -10.96717,6.95467 -10.85492,12.75434 0.1366,7.54733 3.47685,14.48175 10.73818,16.7406 8.03172,2.70257 17.22586,1.93392 24.66094,-2.55921 6.14217,-4.7856 6.71028,-13.6958 3.49781,-20.4765 -1.44495,-3.73134 -4.56003,-6.47068 -8.78786,-6.45923 v 13.28344 H 166.40691 V 131.3289 h 36.10126 v 10.28372 l -9.88833,0.0915 c 9.22745,5.26008 10.1733,16.6128 9.70037,24.48894 0.3325,13.5299 -13.03437,24.56116 -26.292,26.03608 -4.79652,0.35961 -8.85473,0.37853 -15.23296,0.12667 z"
+     style="fill:#666666;fill-opacity:1;stroke-width:0.735607" /><path
+     sodipodi:nodetypes="ccccc"
+     id="path1925-1-7"
+     d="m 135.00354,125.64811 -0.0758,-15.00768 h 67.67588 l -0.008,15.07995 -67.59231,-0.0723 z"
+     style="fill:#d8382d;fill-opacity:1;stroke-width:0.735607" /><path
+     sodipodi:nodetypes="ccccccccccccc"
+     id="path1917-90-1"
+     d="M 160.79658,105.36859 V 89.307826 H 134.5599 v -13.97654 h 26.23668 v -18.02238 h 15.69293 v 18.02238 h 26.11406 v 13.97654 h -26.11406 l -0.25871,16.027244 -15.43422,0.0335 z"
+     style="fill:#40bf4c;fill-opacity:1;stroke-width:0.735607" /><path
+     style="fill:#40bf4c;fill-opacity:1;fill-rule:evenodd;stroke:none;stroke-width:0.735607"
+     d="M 91.825744,18.068854 76.171139,39.953171 h 8.68936 c 0.01763,2.418934 0.222525,4.801478 0.574654,7.126197 l 13.81566,-2.091884 c -0.247091,-1.644053 -0.396048,-3.325842 -0.413778,-5.034313 h 8.643395 z M 100.3312,49.906858 86.975373,53.92971 c 0.679483,2.253194 1.505492,4.445378 2.482673,6.551503 l 12.666194,-5.861871 c -0.70324,-1.517249 -1.30665,-3.088285 -1.79304,-4.712484 z m 4.18377,9.011189 -11.746687,7.562963 c 5.285482,8.195887 12.907037,14.747249 21.930287,18.689025 l 5.58598,-12.804167 c -6.49673,-2.835317 -11.97061,-7.5448 -15.76958,-13.447821 z m 20.41311,15.07995 -3.77,13.470807 c 2.13297,0.596378 4.31508,1.045614 6.55155,1.356282 L 129.6406,74.98647 c -1.60694,-0.22606 -3.17878,-0.559429 -4.71252,-0.988473 z"
+     id="path2836-9-1" /><path
+     style="fill:#40bf4c;fill-opacity:1;fill-rule:evenodd;stroke:none;stroke-width:0.735607"
+     d="m 245.29182,18.068854 -15.65468,21.884317 h 8.68936 c -0.0176,1.708471 -0.16669,3.39026 -0.41378,5.034313 l 13.81566,2.091884 c 0.35214,-2.324719 0.55679,-4.707263 0.57466,-7.126197 h 8.64339 z m -8.45948,31.838004 c -0.48638,1.624199 -1.08981,3.195235 -1.79305,4.712484 l 12.66621,5.861871 c 0.9771,-2.106125 1.80319,-4.298309 2.48267,-6.551503 z m -4.18377,9.011189 c -3.79897,5.903021 -9.27284,10.612504 -15.76958,13.447821 l 5.58598,12.804167 c 9.02325,-3.941776 16.64481,-10.493138 21.93029,-18.689025 z m -20.4131,15.07995 c -1.53374,0.429044 -3.10559,0.762413 -4.71252,0.988473 l 1.93097,13.838616 c 2.23647,-0.310668 4.41864,-0.759904 6.55153,-1.356282 z"
+     id="path3649-8-1" /></svg>
diff --git a/doc/logo.mdwn b/doc/logo.mdwn
index 9055917996..ae006ce3a0 100644
--- a/doc/logo.mdwn
+++ b/doc/logo.mdwn
@@ -4,6 +4,12 @@ Variants of the git-annex logo.
 
 [[logo.svg]]
 
+[[logo-with-cli.svg]]
+
+[[logo-with-cli.png]]
+
+(The font of the `> git annex` text is [JetBrains Mono](https://www.jetbrains.com/lp/mono/))
+
 [[logo-old.png]]
 
 [[logo-old_small.png]]
@@ -16,6 +22,7 @@ Variants of the git-annex logo.
 
 	Copyright: 2007 Henrik Nyh <http://henrik.nyh.se/>
 	           2010 Joey Hess <id@joeyh.name>
-		   2013 John Lawrence
+	           2013 John Lawrence
+	           2024 Yann Büchau <nobodyinperson at posteo de>
 	License: other
 	  Free to modify and redistribute with due credit, and obviously free to use.

expand on what i feel a tweak could help
diff --git a/doc/bugs/unlocked_files_feel_too_slow_.mdwn b/doc/bugs/unlocked_files_feel_too_slow_.mdwn
index 629c28949a..cf9ba5162f 100644
--- a/doc/bugs/unlocked_files_feel_too_slow_.mdwn
+++ b/doc/bugs/unlocked_files_feel_too_slow_.mdwn
@@ -26,7 +26,9 @@ It's not quite clear to me how to reproduce this from scratch. I guess convert a
 
 ### Please provide any additional information below.
 
-Not sure what could help here.
+Not sure what could help here. Maybe having more feedback on what's actually happening in those minutes-long stretches would help. Git, for example, has a percentage/object count thing that pops up when things like `git status` take too long. I would love to see something similar here.
+
+Also, I understand the [[todo/git_smudge_clean_interface_suboptiomal]] problem, and this might just be a specific manifestation of it. It still seems to me having some feedback, if even around the git filter boundaries, would alleviate some of those problems.
 
 ### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
 

diff --git a/doc/bugs/unlocked_files_feel_too_slow_.mdwn b/doc/bugs/unlocked_files_feel_too_slow_.mdwn
new file mode 100644
index 0000000000..629c28949a
--- /dev/null
+++ b/doc/bugs/unlocked_files_feel_too_slow_.mdwn
@@ -0,0 +1,33 @@
+### Please describe the problem.
+
+I recently switched a large (~6TiB) video repository from "normal" to "unlocked" mode, mainly because some external tool was getting confused by symlinks.
+
+As a result, all remotes related to that repositories have this painful, multi-hour process by which they convert their work tree to follow the new layout. That, in itself, is disturbingly slow, with little progress shown to the user. In fact, I often fear that git-annex is messing up and actually adding files as git blobs itself, which makes me interrupt it and start over again, which, obviously, makes things much worse.
+
+But there are other weird things. At the moment, I'm copying files from my laptop with:
+
+    git annex copy --from angela -J2
+
+Being worried about the lack of progress, I interrupted it and ran instead:
+
+    git annex copy --from angela -J2 --not --in here --in angela
+
+That command generated zero output for 12 minutes straight. Looking at strace, it seemed to have read a large number of files in `.git/annex/objects`, which is slow.
+
+Another example is, after the above command completed, i ran `git annex sync -g`, which i expected to take a handful of seconds, but the first line was a `commit` that took nearly a minute.
+
+### What steps will reproduce the problem?
+
+It's not quite clear to me how to reproduce this from scratch. I guess convert an existing large repo?
+
+### What version of git-annex are you using? On what operating system?
+
+10.20230802-1~bpo12+1 on debian bookworm.
+
+### Please provide any additional information below.
+
+Not sure what could help here.
+
+### Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
+
+I use git-annex daily, and it's a lifesaver. :)

Add annextimelog project
diff --git a/doc/projects/annextimelog.mdwn b/doc/projects/annextimelog.mdwn
new file mode 100644
index 0000000000..031032b84b
--- /dev/null
+++ b/doc/projects/annextimelog.mdwn
@@ -0,0 +1,10 @@
+[annextimelog](https://gitlab.com/nobodyinperson/annextimelog) (`atl`) is a cli time tracker and logbook made by [[users/nobodyinperson]]. It uses git-annex to store and sync data. `atl` has a loosely similar usage to [timewarrior](https://timewarrior.net/) but improves on many of its pain points:
+
+- tracking of simultaneous events
+- conflict-free syncing
+- more powerful tagging system
+- more powerful search
+- more output formats (JSON, hledger timeclock, ...)
+- a more sophisticated event specification language
+
+annextimelog is also an experiment how useable git-annex is as a backend for syncable application data storage. While first sounding silly, it works very well.

add openneuro project
diff --git a/doc/projects/openneuro.mdwn b/doc/projects/openneuro.mdwn
new file mode 100644
index 0000000000..e822f437c6
--- /dev/null
+++ b/doc/projects/openneuro.mdwn
@@ -0,0 +1,28 @@
+OpenNeuro ([openneuro.org](https://openneuro.org))
+==================================================
+
+[OpenNeuro](https://openneuro.org) is a free and open platform for sharing [BIDS](https://bids.neuroimaging.io)-compliant neuroimaging (MRI, PET, MEG, EEG, and iEEG) data. 
+Git-annex is the core tool for data logistics at OpenNeuro, e.g. for exporting the content to AWS S3 into open `openneuro.org` S3 bucket, and providing all datasets as DataLad datasets (git/git-annex repositories with a unique DataLad ID) from the [github.com/OpenNeuroDatasets](https://github.com/OpenNeuroDatasets) organization.
+
+## TODOs
+
+[[!inline pages="todo/* and !todo/done and !link(todo/done) and tagged(projects/openneuro)" sort=mtime feeds=no actions=yes archive=yes show=0 template=buglist]] 
+
+<details>
+<summary>Done</summary>
+
+[[!inline pages="todo/* and !todo/done and link(todo/done) and tagged(projects/openneuro)" sort=mtime feeds=no actions=yes archive=yes show=0 template=buglist]] 
+
+</details>
+
+## BUGs
+
+[[!inline pages="bugs/* and !bugs/done and !link(bugs/done) and tagged(projects/openneuro)" sort=mtime feeds=no actions=yes archive=yes show=0 template=buglist]] 
+
+<details>
+<summary>Done</summary>
+
+[[!inline pages="((bugs/* and !bugs/done and link(bugs/done)) or projects/openneuro/bugs-done/*) and tagged(projects/openneuro)" sort=mtime feeds=no actions=yes archive=yes show=0 template=buglist]] 
+
+</details>
+

comment
diff --git a/doc/forum/Git_Annex_shirts_at_hellotux.com/comment_1_fa72689070c39054d166d32fa144fabc._comment b/doc/forum/Git_Annex_shirts_at_hellotux.com/comment_1_fa72689070c39054d166d32fa144fabc._comment
new file mode 100644
index 0000000000..7be35c8765
--- /dev/null
+++ b/doc/forum/Git_Annex_shirts_at_hellotux.com/comment_1_fa72689070c39054d166d32fa144fabc._comment
@@ -0,0 +1,10 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 1"""
+ date="2024-02-16T18:10:20Z"
+ content="""
+I did get back to that email today.
+
+For what it's worth, I'm generally ok with anyone making git-annex merch. 
+See the [[logo]] page for its license.
+"""]]

Added a comment
diff --git a/doc/bugs/identically_configured_remotes_behave_differently/comment_1_14fa5263ee9feedff8c36f86e4c33fd8._comment b/doc/bugs/identically_configured_remotes_behave_differently/comment_1_14fa5263ee9feedff8c36f86e4c33fd8._comment
new file mode 100644
index 0000000000..40bbc863db
--- /dev/null
+++ b/doc/bugs/identically_configured_remotes_behave_differently/comment_1_14fa5263ee9feedff8c36f86e4c33fd8._comment
@@ -0,0 +1,351 @@
+[[!comment format=mdwn
+ username="jstritch"
+ avatar="http://cdn.libravatar.org/avatar/56756b34ff409071074762951d270002"
+ subject="comment 1"
+ date="2024-02-16T17:48:55Z"
+ content="""
+I have attached a script and log file demonstrating the problem. The script:
+
+1) sets up an integration repository and two backup repositories each containing one file.
+
+2) adds another file to the integration repository, pushes it to both backups, and does a `git annex list` which shows correctly.
+
+3) does a `git annex sync music-backup-one` to introduce the problem.
+
+4) adds another file to the integration repository, pushes it to both backups, and does a `git annex list` which shows the file missing from `music-backup-one`.
+
+5) does `git annex mirror . --to=music-backup-one` then another `git annex list` which shows the files as expected.
+
+If the sync command is commented out, the last two lines are unnecessary. The sync command should not change the behavior of other commands.
+
+[[!format sh \"\"\"
+mkdir music-repo
+mkdir music-backup-one
+mkdir music-backup-two
+
+cd music-repo
+git init --initial-branch main
+git annex init music-repo
+git config annex.adjustedbranchrefresh true
+git config receive.denyCurrentBranch updateInstead
+git annex config --set annex.largefiles include=*.xtx
+git annex group here manual
+git annex wanted here standard
+echo \"test file one\" >> test-file-one.xtx
+git annex add .
+git commit -m \"add test file one\"
+git annex adjust --unlock
+
+cd ../music-backup-one
+git clone --origin music-repo ../music-repo .
+git annex init music-backup-one
+cd ../music-repo
+git remote add music-backup-one ../music-backup-one
+git annex push music-backup-one
+
+cd ../music-backup-two
+git clone --origin music-repo ../music-repo .
+git annex init music-backup-two
+cd ../music-repo
+git remote add music-backup-two ../music-backup-two
+git annex push music-backup-two
+
+cd ../music-backup-one
+git config annex.adjustedbranchrefresh true
+git config receive.denyCurrentBranch updateInstead
+git annex config --set annex.largefiles include=*.xtx
+git annex group here manual
+git annex wanted here standard
+git annex adjust --lock
+
+cd ../music-backup-two
+git config annex.adjustedbranchrefresh true
+git config receive.denyCurrentBranch updateInstead
+git annex config --set annex.largefiles include=*.xtx
+git annex group here manual
+git annex wanted here standard
+git annex adjust --lock
+
+cd ../music-repo
+echo \"test file two\" >> test-file-two.xtx
+git annex add .
+git commit -m \"add test file two\"
+git annex push music-backup-one
+git annex push music-backup-two
+git annex list
+
+# *** comment out line below to fix ***
+git annex sync music-backup-one
+
+echo \"test file three\" >> test-file-three.xtx
+git annex add .
+git commit -m \"add test file three\"
+git annex push music-backup-one
+git annex push music-backup-two
+git annex list
+
+# *** lines below only needed if sync is not commented out ***
+git annex mirror . --to=music-backup-one
+git annex list
+\"\"\"]]
+
+Here's the log file:
+
+[[!format sh \"\"\"
+Initialized empty Git repository in /home/juanito/Documents/music-repo/.git/
+init music-repo ok
+(recording state in git...)
+annex.largefiles include=*.xtx ok
+(recording state in git...)
+group here ok
+(recording state in git...)
+wanted here ok
+(recording state in git...)
+add test-file-one.xtx 
+ok                                
+(recording state in git...)
+[main (root-commit) 3358556] add test file one
+ 1 file changed, 1 insertion(+)
+ create mode 120000 test-file-one.xtx
+adjust  
+Switched to branch 'adjusted/main(unlocked)'
+ok
+Cloning into '.'...
+done.
+init music-backup-one (merging music-repo/git-annex into git-annex...)
+ok
+(recording state in git...)
+copy test-file-one.xtx (to music-backup-one...) 
+(recording state in git...)       
+ok
+(recording state in git...)
+push music-backup-one 
+Enumerating objects: 9, done.
+Counting objects: 100% (9/9), done.
+Delta compression using up to 12 threads
+Compressing objects: 100% (4/4), done.
+Writing objects: 100% (5/5), 422 bytes | 422.00 KiB/s, done.
+Total 5 (delta 2), reused 0 (delta 0), pack-reused 0
+To ../music-backup-one
+ * [new branch]      main -> synced/main
+ * [new branch]      git-annex -> synced/git-annex
+ok
+Cloning into '.'...
+done.
+init music-backup-two (merging music-repo/git-annex into git-annex...)
+(recording state in git...)
+ok
+(recording state in git...)
+copy test-file-one.xtx (to music-backup-two...) 
+(recording state in git...)       
+ok
+(recording state in git...)
+push music-backup-two 
+Enumerating objects: 9, done.
+Counting objects: 100% (9/9), done.
+Delta compression using up to 12 threads
+Compressing objects: 100% (4/4), done.
+Writing objects: 100% (5/5), 447 bytes | 447.00 KiB/s, done.
+Total 5 (delta 2), reused 0 (delta 0), pack-reused 0
+To ../music-backup-two
+ * [new branch]      main -> synced/main
+ * [new branch]      git-annex -> synced/git-annex
+ok
+annex.largefiles include=*.xtx (merging synced/git-annex into git-annex...)
+(recording state in git...)
+ok
+group here ok
+(recording state in git...)
+wanted here ok
+(recording state in git...)
+adjust  
+Switched to branch 'adjusted/main(locked)'
+ok
+annex.largefiles include=*.xtx (merging synced/git-annex into git-annex...)
+(recording state in git...)
+ok
+group here ok
+(recording state in git...)
+wanted here ok
+(recording state in git...)
+adjust  
+Switched to branch 'adjusted/main(locked)'
+ok
+add test-file-two.xtx 
+ok                                
+(recording state in git...)
+[adjusted/main(unlocked) e5a8b04] add test file two
+ 1 file changed, 1 insertion(+)
+ create mode 100644 test-file-two.xtx
+copy test-file-two.xtx (to music-backup-one...) 
+ok                                
+(recording state in git...)
+push music-backup-one 
+Enumerating objects: 23, done.
+Counting objects: 100% (23/23), done.
+Delta compression using up to 12 threads
+Compressing objects: 100% (15/15), done.
+Writing objects: 100% (18/18), 1.49 KiB | 1.49 MiB/s, done.
+Total 18 (delta 6), reused 0 (delta 0), pack-reused 0
+remote: merge synced/main (Merging into main...) 
+remote: Updating 3358556..e4f11f9
+remote: Fast-forward
+remote:  test-file-two.xtx | 1 +
+remote:  1 file changed, 1 insertion(+)

(Diff truncated)
removed
diff --git a/doc/bugs/identically_configured_remotes_behave_differently/comment_1_d69d6a014c7265fb901dc407516e232b._comment b/doc/bugs/identically_configured_remotes_behave_differently/comment_1_d69d6a014c7265fb901dc407516e232b._comment
deleted file mode 100644
index 2677b04dab..0000000000
--- a/doc/bugs/identically_configured_remotes_behave_differently/comment_1_d69d6a014c7265fb901dc407516e232b._comment
+++ /dev/null
@@ -1,8 +0,0 @@
-[[!comment format=mdwn
- username="jstritch"
- avatar="http://cdn.libravatar.org/avatar/56756b34ff409071074762951d270002"
- subject="comment 1"
- date="2024-02-06T19:54:40Z"
- content="""
-This is possibly another manifestation of `pull doesn't work on adjusted branches`. I'll await your reply to it.
-"""]]

Fix typo
diff --git a/doc/forum/git_annex_lock__47__unlock__47__fix_on_non-committed_files.mdwn b/doc/forum/git_annex_lock__47__unlock__47__fix_on_non-committed_files.mdwn
index ec8f1f08fe..3e7c562114 100644
--- a/doc/forum/git_annex_lock__47__unlock__47__fix_on_non-committed_files.mdwn
+++ b/doc/forum/git_annex_lock__47__unlock__47__fix_on_non-committed_files.mdwn
@@ -4,7 +4,7 @@ In that case, `git annex unlock` refuses to work:
 
 ```
 mv file file_version1
-git annex unlock file
+git annex unlock file_version1
 error: pathspec 'file' did not match any file(s) known to git
 Did you forget to 'git add'?
 unlock: 1 failed

diff --git a/doc/forum/git_annex_lock__47__unlock__47__fix_on_non-committed_files.mdwn b/doc/forum/git_annex_lock__47__unlock__47__fix_on_non-committed_files.mdwn
new file mode 100644
index 0000000000..ec8f1f08fe
--- /dev/null
+++ b/doc/forum/git_annex_lock__47__unlock__47__fix_on_non-committed_files.mdwn
@@ -0,0 +1,15 @@
+In an ideal world, one could just use `git annex diffdriver` to compare annex'ed files. However, sometimes one might need an approach that is constructed simpler. For example, I want to run a comparison utility that is in a container which translates my paths for better reproducibility of scientific results. Or I have a colleage who is not familiar with all the intricacies of git annex and knows git just enough `git checkout` version 1, move it, the `git checkout` the second version and compare these two. Then the situation can arise that one has a symlink to `.git/annex/objects/...` which is named differently than what is saved in the tree. 
+
+In that case, `git annex unlock` refuses to work:
+
+```
+mv file file_version1
+git annex unlock file
+error: pathspec 'file' did not match any file(s) known to git
+Did you forget to 'git add'?
+unlock: 1 failed
+```
+
+Similar story goes with `git lock` and `git fix`. I think it would be a nice behaviour if these commands would just do their work on the new file name. For example, git unlock would replace a symlink with its content even if the file is not committed with this name. 
+
+Is it possible to implement this in git annex?

Added a comment: a script to hide history
diff --git a/doc/tips/redacting_history_by_converting_git_files_to_annexed/comment_3_7e92db3d8a392e43b305fa8ecc0e39d3._comment b/doc/tips/redacting_history_by_converting_git_files_to_annexed/comment_3_7e92db3d8a392e43b305fa8ecc0e39d3._comment
new file mode 100644
index 0000000000..404146779a
--- /dev/null
+++ b/doc/tips/redacting_history_by_converting_git_files_to_annexed/comment_3_7e92db3d8a392e43b305fa8ecc0e39d3._comment
@@ -0,0 +1,39 @@
+[[!comment format=mdwn
+ username="yarikoptic"
+ avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
+ subject="a script to hide history"
+ date="2024-02-15T22:05:12Z"
+ content="""
+Here is a script I crafted to use to make it easy and reuse current tree object for new \"squashed history\" commit
+
+```bash
+#!/bin/bash
+#
+# A helper to establish an alternative history to hide commits which could have
+# leaked personal data etc.
+#
+# More information on motivation etc and another implementation could be
+# found at https://git-annex.branchable.com/tips/redacting_history_by_converting_git_files_to_annexed/
+#
+
+set -eu
+
+BRANCH=$(git rev-parse --abbrev-ref HEAD)
+: \"${SECRET_BRANCH:=unredacted-$BRANCH}\"
+SAFE_BASE=\"$1\"
+
+git branch \"${SECRET_BRANCH}\"
+
+rm -f .git/COMBINED_COMMIT_MESSAGE
+echo -e \"Combined commits to hide away sensitive data\n\" >> .git/COMBINED_COMMIT_MESSAGE
+git log --format=%B \"$SAFE_BASE..HEAD\" >> .git/COMBINED_COMMIT_MESSAGE
+
+# the tree we are on ATM
+TREE_HASH=$(git log -1 --format=%T HEAD)
+NEW_COMMIT=$(git commit-tree $TREE_HASH -p \"$SAFE_BASE\" -F .git/COMBINED_COMMIT_MESSAGE)
+rm -f .git/COMBINED_COMMIT_MESSAGE
+git reset --hard $NEW_COMMIT
+
+git replace \"$BRANCH\" \"$SECRET_BRANCH\"
+```
+"""]]

update thanks and separate out 2023 name list
diff --git a/doc/thanks.mdwn b/doc/thanks.mdwn
index 9240246c72..aa0d090b4c 100644
--- a/doc/thanks.mdwn
+++ b/doc/thanks.mdwn
@@ -31,6 +31,20 @@ tips, user support, etc. Two names that come to mind are Anarcat and
 CandyAngel. John Lawrence made the logo. And many others have
 contributed good bug reports and great ideas.
 
+## financial support, 2024
+
+git-annex development is supported in large part by:
+
+* [DANDI](https://www.dandiarchive.org/), funded by
+  [a NIH grant](https://projectreporter.nih.gov/project_info_description.cfm?aid=9981835)
+  awarded to MIT, Dartmouth College, and Kitware.
+* [ReproNim](https://repronim.org/), funded by
+  [a NIH grant](https://projectreporter.nih.gov/project_info_details.cfm?aid=8999833)
+  awarded to UMass Medical School Worcester, Dartmouth College, MIT, et al.
+
+Thanks also to these folks for their support: 
+[[!inline raw=yes pages="thanks/list"]] and anonymous supporters.
+
 ## financial support, 2019-2023
 
 <img alt="McGill logo" src="https://mcgill.ca/hbhl/sites/all/themes/moriarty/images/logo-red.svg" width=100>
@@ -56,7 +70,7 @@ git-annex development was supported in large part by:
 * Gioele Barabucci ENK
 
 Thanks also to these folks for their support: 
-[[!inline raw=yes pages="thanks/list"]] and anonymous supporters.
+[[!inline raw=yes pages="thanks/list.2023"]] and anonymous supporters.
 
 ## financial support, 2014-2018
 
diff --git a/doc/thanks/list b/doc/thanks/list
index a549f5fc64..f64604c81f 100644
--- a/doc/thanks/list
+++ b/doc/thanks/list
@@ -1,109 +1,77 @@
-Ethan Aubin, 
-Thomas May, 
-Jake Vosloo, 
-Trenton Cronholm, 
+Kanak Kshetri, 
+Andrew Gilbert, 
+Noam Kremen, 
+Mark Reidenbach, 
+Martin D, 
+James Greenaway, 
+Timothy Schumann, 
+Graham Spencer, 
+sirmacik, 
+Cesar, 
+Rian McGuire, 
+Jelmer Vernooij, 
+Svenne Krap, 
+Ryan Rix, 
+James Read, 
+Luke Shumaker, 
+EVAN HENSHAWPLATH, 
+Nikhil, 
+Josh Moller-Mara, 
+André Pereira, 
+Brennen Bearnes, 
+N. A., 
 Denis Dzyubenko, 
-Thomas Koch, 
-mo, 
-Jochen Bartl, 
-Brock Spratlen, 
-Jack Hill, 
+paul walmsley, 
+Pluralist Extremist, 
+Falk Hüffner, 
 Nick Piper, 
-Eric Drechsel, 
 Brett Eisenberg, 
+Eric Prestemon, 
 John Pellman, 
-Boyd Stephen Smith Jr., 
-Ilya Shlyakhter, 
 Markus Hauru, 
-Kuno Woudt, 
 Ewen McNeill, 
-FBC, 
-Amitai Schleier, 
-Michal Sojka, 
-Brennen Bearnes, 
-Eric Prestemon, 
-BonusWavePilot, 
-Baldur Kristinsson, 
-Matthias Urlichs, 
-Diederik de Haas, 
-Boris, 
-Jason Woofenden, 
-William Hay, 
-Nathan Howell, 
-Josh, 
+Nicolas Pouillard, 
+Johannes Grødem, 
+Gioele Barabucci, 
+TD, 
+Erik Bjäreholt, 
 Silvio Ankermann, 
-John Carr, 
-Henrik Riomar, 
-Eskild Hustvedt, 
-Nick Daly, 
-Daniel Takamori, 
-John Lee, 
+Michael Alan Dorman, 
 Lars Wallenborn, 
-Willard Korfhage, 
-Erik Bjäreholt, 
-Calvin Beck, 
-Johannes Grødem, 
+John Kozak, 
 Michael Tolly, 
-AlexS, 
-paul walmsley, 
-tdittr, 
-Peter, 
-Fernando Jimenez, 
-Sergey Karpukhin, 
-Jeff Moe, 
-Alexander Thompson, 
-Nicolas Pouillard, 
-Tyler Cipriani, 
-Josh Moller-Mara, 
-Lp and Nick Daly, 
-Walltime, 
-Caleb Allen, 
-TD, 
-Pedro Araújo, 
-Ryan Newton, 
-David W, 
-L N D, 
-EVAN HENSHAWPLATH, 
-James Read, 
-Luke Shumaker, 
-Marius Konitzer, 
-Ryan Rix, 
-Svenne Krap, 
-Jelmer Vernooij, 
-Rian McGuire, 
-Cesar, 
-LND, 
-sirmacik, 
-Abdó Roig-Maranges, 
+Daniel Takamori, 
+Jack Hill, 
 Alex Watson, 
-allanfranta, 
-André Pereira, 
+William Hay, 
+Michal Sojka, 
+Rob Hunter, 
+Boris, 
+Trenton Cronholm, 
 E. Dunham, 
-Falk Hüffner, 
+Jake Vosloo, 
+Brock Spratlen, 
 Filippo Giunchedi, 
-Gioele Barabucci, 
-John Kozak, 
+LND, 
 Karl Semich, 
-Madison McGaffin, 
-Marco Roeland, 
+Boyd Stephen Smith Jr., 
+Jochen Bartl, 
+Nathan Howell, 
+Kuno Woudt, 
+Baldur Kristinsson, 
+BonusWavePilot, 
 Mark Eichin, 
-Michael Alan Dorman, 
-N. A., 
-Nikhil, 
-Rob Hunter, 
+FBC, 
+Marco Roeland, 
+Josh, 
+visnescire, 
 Simon Michael, 
-Stephan Burkhardt, 
 Thomas, 
-visnescire, 
-Graham Spencer, 
-Timothy Schumann, 
-Mark Reidenbach, 
-Martin D, 
-James Greenaway, 
-Kanak Kshetri, 
-Andrew Gilbert, 
-Noam Kremen, 
-Pluralist Extremist, 
+Henrik Riomar, 
+allanfranta, 

(Diff truncated)
Suggest cooperating with hellotux.com for git-annex-branded shirts
diff --git a/doc/forum/Git_Annex_shirts_at_hellotux.com.mdwn b/doc/forum/Git_Annex_shirts_at_hellotux.com.mdwn
new file mode 100644
index 0000000000..d6c3357d3e
--- /dev/null
+++ b/doc/forum/Git_Annex_shirts_at_hellotux.com.mdwn
@@ -0,0 +1,3 @@
+The fine people over at https://hellotux.com would love to make git-annex branded shirts, hoodies and currently also backpacks with git-annex branding. We recently started a cooperation for DataLad shirts, which are now available [here](https://www.hellotux.com/datalad). I have one of those and they look and feel very good. I would love to have the same with git annex's logo embroidered on it. Especially as the distribits meeting is upcoming, now would be a good time to give Gabor from hellotux a quick go that he may start. It won't cost git-annex anything, for datalad the process was very simple: send Gabor the SVG, they make a test stitch and if that looks fine, he'll add it to the webpage.
+
+@joey if you're against this for some reason, you may just say so, then I'll stop bringing this up but I'd like to know what you think of it. 😃

pre-commit: Avoid committing the git-annex branch
Except when a commit is made in a view, which changes metadata.
Make the assistant commit the git-annex branch after git commit of working
tree changes.
This allows using the annex.commitmessage-command in the assistant to
generate a commit message for the git-annex branch that relies on state
gathered during the commit of the working tree. Eg, it might reuse the
commit message.
Note that, when not using the assistant, a git-annex add still commits
the git-annex branch, so such a annex.commitmessage-command set up would
not work then. But if someone is using the assistant and wants
programmatic control over commit messages, this is useful. Someone not
using the assistant can get the same result by using annex.alwayscommit=false
during the git-annex add, and git-annex merge after they git commit.
pre-commit was never really intended to commit the git-annex branch
(except after recording changed metadata), but the assistant did sort of
rely on it. It does later commit the git-annex branch before pushing to
remotes, but I didn't want to risk building up lots of uncommitted changes
to it if that didn't happen frequently.
Sponsored-by: the NIH-funded NICEMAN (ReproNim TR&D3) project
diff --git a/Assistant/Threads/Committer.hs b/Assistant/Threads/Committer.hs
index 79eed77e95..07013c0486 100644
--- a/Assistant/Threads/Committer.hs
+++ b/Assistant/Threads/Committer.hs
@@ -1,6 +1,6 @@
 {- git-annex assistant commit thread
  -
- - Copyright 2012-2023 Joey Hess <id@joeyh.name>
+ - Copyright 2012-2024 Joey Hess <id@joeyh.name>
  -
  - Licensed under the GNU AGPL version 3 or higher.
  -}
@@ -19,7 +19,6 @@ import Assistant.TransferQueue
 import Assistant.Drop
 import Types.Transfer
 import Logs.Location
-import qualified Annex.Queue
 import Utility.ThreadScheduler
 import qualified Utility.Lsof as Lsof
 import qualified Utility.DirWatcher as DirWatcher
@@ -35,6 +34,8 @@ import Annex.InodeSentinal
 import Annex.CurrentBranch
 import Annex.FileMatcher
 import qualified Annex
+import qualified Annex.Queue
+import qualified Annex.Branch
 import Utility.InodeCache
 import qualified Database.Keys
 import qualified Command.Sync
@@ -248,6 +249,11 @@ commitStaged msg = do
 				]
 			when ok $
 				Command.Sync.updateBranches =<< getCurrentBranch
+			{- Commit the git-annex branch. This comes after
+			 - the commit of the staged changes, so that
+			 - annex.commitmessage-command can examine that
+			 - commit. -}
+			Annex.Branch.commit =<< Annex.Branch.commitMessage
 			return ok
 
 {- If there are PendingAddChanges, or InProcessAddChanges, the files
diff --git a/CHANGELOG b/CHANGELOG
index e4d178c408..59efa6a74e 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -12,6 +12,8 @@ git-annex (10.20240130) UNRELEASED; urgency=medium
   * assistant, undo: When committing, let the usual git commit
     hooks run.
   * Added annex.commitmessage-command config.
+  * pre-commit: Avoid committing the git-annex branch
+    (except when a commit is made in a view, which changes metadata).
 
  -- Joey Hess <id@joeyh.name>  Mon, 29 Jan 2024 15:59:33 -0400
 
diff --git a/Command/PreCommit.hs b/Command/PreCommit.hs
index d8fdeea197..bc69e4a210 100644
--- a/Command/PreCommit.hs
+++ b/Command/PreCommit.hs
@@ -1,6 +1,6 @@
 {- git-annex command
  -
- - Copyright 2010-2014 Joey Hess <id@joeyh.name>
+ - Copyright 2010-2024 Joey Hess <id@joeyh.name>
  -
  - Licensed under the GNU AGPL version 3 or higher.
  -}
@@ -20,12 +20,14 @@ import Logs.View
 import Logs.MetaData
 import Types.View
 import Types.MetaData
+import qualified Annex
+import qualified Annex.Branch
 
 import qualified Data.Set as S
 import qualified Data.Text as T
 
 cmd :: Command
-cmd = command "pre-commit" SectionPlumbing
+cmd = noCommit $ command "pre-commit" SectionPlumbing
 	"run by git pre-commit hook"
 	paramPaths
 	(withParams seek)
@@ -47,9 +49,14 @@ seek ps = do
 	-- committing changes to a view updates metadata
 	currentView >>= \case
 		Nothing -> noop
-		Just (v, _madj) -> withViewChanges
-			(addViewMetaData v)
-			(removeViewMetaData v)
+		Just (v, _madj) -> do
+			withViewChanges
+				(addViewMetaData v)
+				(removeViewMetaData v)
+			-- Manually commit in this case, because
+			-- noCommit prevents automatic commit.
+			whenM (annexAlwaysCommit <$> Annex.getGitConfig) $
+				Annex.Branch.commit =<< Annex.Branch.commitMessage
 
 addViewMetaData :: View -> ViewedFile -> Key -> CommandStart
 addViewMetaData v f k = starting "metadata" ai si $
diff --git a/doc/todo/make_annex___34__respect__34___.git__47__hooks__47__prepare-commit-msg.mdwn b/doc/todo/make_annex___34__respect__34___.git__47__hooks__47__prepare-commit-msg.mdwn
index 37af943de4..44b9e7c603 100644
--- a/doc/todo/make_annex___34__respect__34___.git__47__hooks__47__prepare-commit-msg.mdwn
+++ b/doc/todo/make_annex___34__respect__34___.git__47__hooks__47__prepare-commit-msg.mdwn
@@ -134,3 +134,5 @@ Date:   Fri Feb 2 12:06:41 2024 -0500
 
 [[!meta author=yoh]]
 [[!tag projects/repronim]]
+
+> [[done]] --[[Joey]]
diff --git a/doc/todo/make_annex___34__respect__34___.git__47__hooks__47__prepare-commit-msg/comment_6_b8bb9e5c7156e4defd7a2e8b7777e1ba._comment b/doc/todo/make_annex___34__respect__34___.git__47__hooks__47__prepare-commit-msg/comment_6_b8bb9e5c7156e4defd7a2e8b7777e1ba._comment
index 4aa985a935..257e7afd4f 100644
--- a/doc/todo/make_annex___34__respect__34___.git__47__hooks__47__prepare-commit-msg/comment_6_b8bb9e5c7156e4defd7a2e8b7777e1ba._comment
+++ b/doc/todo/make_annex___34__respect__34___.git__47__hooks__47__prepare-commit-msg/comment_6_b8bb9e5c7156e4defd7a2e8b7777e1ba._comment
@@ -3,16 +3,34 @@
  subject="""comment 6"""
  date="2024-02-12T17:36:32Z"
  content="""
-Turns out that the assistant doesn't commit to the git-annex branch itself,
-instead the pre-commit hook runs `git-annex pre-commit`, and 
-the git-annex branch commit on process shutdown is where the commit
-happens.
+I've implemented annex.commitmessage-command and made sure that the
+assistant calls it after committing the working tree changes.
 
-A bit surprising! If the pre-commit hook didn't run git-annex, 
-the assistant would later explicitly commit the branch before
-pushing to remotes.
+Example of using the prepare-commit-msg hook to generate a commit message
+and then reusing that message for the git-annex branch commit:
 
-Anyway, this does mean you can rely on the git-annex branch commit
-happening after the working tree commit. At least, when there are
-no other git-annex processes running.
+	joey@darkstar:~/tmp/a>cat .git/hooks/prepare-commit-msg
+	#!/bin/sh
+	msg="files: $(git diff --name-only --cached)"
+	echo $msg > .git/last-commit-msg
+	echo $msg > $1
+	
+	joey@darkstar:~/tmp/a>git config annex.commitmessage-command
+	cat .git/last-commit-msg
+	
+	joey@darkstar:~/tmp/a>git-annex assistant
+	joey@darkstar:~/tmp/a>date > bar
+
+	joey@darkstar:~/tmp/a>git log -n1 master
+	commit 676f87d02f031192c7eb0b7d29a6ce2429b8c727 (HEAD -> master, synced/master)
+	Author: Joey Hess <joeyh@joeyh.name>
+	Date:   Mon Feb 12 14:29:12 2024 -0400
+	
+	    files: bar
+	joey@darkstar:~/tmp/a>git log -n1 git-annex
+	commit 676588dd6d921115782e61be85f75e395cc480b6 (git-annex)
+	Author: Joey Hess <joeyh@joeyh.name>
+	Date:   Mon Feb 12 14:29:12 2024 -0400
+	
+	    files: bar
 """]]

added annex.commitmessage-command config
Sponsored-by: the NIH-funded NICEMAN (ReproNim TR&D3) project
diff --git a/Annex/Branch.hs b/Annex/Branch.hs
index 8f927a78b4..42942d886f 100644
--- a/Annex/Branch.hs
+++ b/Annex/Branch.hs
@@ -500,11 +500,19 @@ append jl f appendable toappend = do
 {- Commit message used when making a commit of whatever data has changed
  - to the git-annex branch. -}
 commitMessage :: Annex String
-commitMessage = fromMaybe "update" . annexCommitMessage <$> Annex.getGitConfig
+commitMessage = fromMaybe "update" <$> getCommitMessage
 
 {- Commit message used when creating the branch. -}
 createMessage :: Annex String
-createMessage = fromMaybe "branch created" . annexCommitMessage <$> Annex.getGitConfig
+createMessage = fromMaybe "branch created" <$> getCommitMessage
+
+getCommitMessage :: Annex (Maybe String)
+getCommitMessage = do
+	config <- Annex.getGitConfig
+	case annexCommitMessageCommand config of
+		Nothing -> return (annexCommitMessage config)
+		Just cmd -> catchDefaultIO (annexCommitMessage config) $
+			Just <$> liftIO (readProcess "sh" ["-c", cmd])
 
 {- Stages the journal, and commits staged changes to the branch. -}
 commit :: String -> Annex ()
diff --git a/CHANGELOG b/CHANGELOG
index 8057fb3c81..e4d178c408 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -11,6 +11,7 @@ git-annex (10.20240130) UNRELEASED; urgency=medium
   * stack.yaml: Update to lts-22.9 and use crypton.
   * assistant, undo: When committing, let the usual git commit
     hooks run.
+  * Added annex.commitmessage-command config.
 
  -- Joey Hess <id@joeyh.name>  Mon, 29 Jan 2024 15:59:33 -0400
 
diff --git a/Types/GitConfig.hs b/Types/GitConfig.hs
index f9e0149ddb..f97ee52192 100644
--- a/Types/GitConfig.hs
+++ b/Types/GitConfig.hs
@@ -88,6 +88,7 @@ data GitConfig = GitConfig
 	, annexAlwaysCommit :: Bool
 	, annexAlwaysCompact :: Bool
 	, annexCommitMessage :: Maybe String
+	, annexCommitMessageCommand :: Maybe String
 	, annexMergeAnnexBranches :: Bool
 	, annexDelayAdd :: Maybe Int
 	, annexHttpHeaders :: [String]
@@ -176,6 +177,7 @@ extractGitConfig configsource r = GitConfig
 	, annexAlwaysCommit = getbool (annexConfig "alwayscommit") True
 	, annexAlwaysCompact = getbool (annexConfig "alwayscompact") True
 	, annexCommitMessage = getmaybe (annexConfig "commitmessage")
+	, annexCommitMessageCommand = getmaybe (annexConfig "commitmessage-command")
 	, annexMergeAnnexBranches = getbool (annexConfig "merge-annex-branches") True
 	, annexDelayAdd = getmayberead (annexConfig "delayadd")
 	, annexHttpHeaders = getlist (annexConfig "http-headers")
diff --git a/doc/git-annex.mdwn b/doc/git-annex.mdwn
index c4bd2b8df3..9d61e675a1 100644
--- a/doc/git-annex.mdwn
+++ b/doc/git-annex.mdwn
@@ -1088,6 +1088,11 @@ repository, using [[git-annex-config]]. See its man page for a list.)
   This works well in combination with annex.alwayscommit=false,
   to gather up a set of changes and commit them with a message you specify.
 
+* `annex.commitmessage-command`
+
+  This command is run and its output is used as the commit message to the
+  git-annex branch.
+
 * `annex.alwayscompact`
 
   By default, git-annex compacts data it records in the git-annex branch.
diff --git a/doc/todo/make_annex___34__respect__34___.git__47__hooks__47__prepare-commit-msg/comment_6_b8bb9e5c7156e4defd7a2e8b7777e1ba._comment b/doc/todo/make_annex___34__respect__34___.git__47__hooks__47__prepare-commit-msg/comment_6_b8bb9e5c7156e4defd7a2e8b7777e1ba._comment
new file mode 100644
index 0000000000..4aa985a935
--- /dev/null
+++ b/doc/todo/make_annex___34__respect__34___.git__47__hooks__47__prepare-commit-msg/comment_6_b8bb9e5c7156e4defd7a2e8b7777e1ba._comment
@@ -0,0 +1,18 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 6"""
+ date="2024-02-12T17:36:32Z"
+ content="""
+Turns out that the assistant doesn't commit to the git-annex branch itself,
+instead the pre-commit hook runs `git-annex pre-commit`, and 
+the git-annex branch commit on process shutdown is where the commit
+happens.
+
+A bit surprising! If the pre-commit hook didn't run git-annex, 
+the assistant would later explicitly commit the branch before
+pushing to remotes.
+
+Anyway, this does mean you can rely on the git-annex branch commit
+happening after the working tree commit. At least, when there are
+no other git-annex processes running.
+"""]]

comment
diff --git a/doc/tips/redacting_history_by_converting_git_files_to_annexed/comment_2_6b682415956756b969767f7de267f1f0._comment b/doc/tips/redacting_history_by_converting_git_files_to_annexed/comment_2_6b682415956756b969767f7de267f1f0._comment
new file mode 100644
index 0000000000..081ca287dc
--- /dev/null
+++ b/doc/tips/redacting_history_by_converting_git_files_to_annexed/comment_2_6b682415956756b969767f7de267f1f0._comment
@@ -0,0 +1,13 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 2"""
+ date="2024-02-12T17:01:13Z"
+ content="""
+> Another note worthwhile making IMHO is that AFAIK those `git replace` markers
+> are local only, and whoever has unredacted-master later on might need to set
+> them up as well for their local clones to make such a \"collapse\" of histories
+
+Right, any repository you fetch unredacted-master into, you will also want to 
+fetch refs/redacted/ to as well, and run git replace there, as shown in the last
+code block of the tip above.
+"""]]

Added a comment
diff --git a/doc/tips/redacting_history_by_converting_git_files_to_annexed/comment_1_aece062800c6ed78988161d40e36bba4._comment b/doc/tips/redacting_history_by_converting_git_files_to_annexed/comment_1_aece062800c6ed78988161d40e36bba4._comment
new file mode 100644
index 0000000000..e8af23f4c9
--- /dev/null
+++ b/doc/tips/redacting_history_by_converting_git_files_to_annexed/comment_1_aece062800c6ed78988161d40e36bba4._comment
@@ -0,0 +1,11 @@
+[[!comment format=mdwn
+ username="yarikoptic"
+ avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
+ subject="comment 1"
+ date="2024-02-11T18:54:45Z"
+ content="""
+Thank you!  In my case, since safe commit is too far in the past, what I had in mind is a little different: I wanted to have a completely disconnected history with a commit which had `$privatefile`s moved to annex, but I think the approach is \"the same\" in effect.
+The only thing I would do differently is to first convert files to git-annex in `master` (to become `unredacted-master`), so I end up with the same tree in `unredacted-master` and `master` happen that later I would need to cherry-pick some changes accidentally committed (e.g. by collaborators) on top of `unredacted-master`.
+
+Another note worthwhile making IMHO is that AFAIK those `git replace` markers are local only, and whoever has unredacted-master later on might need to set them up as well for their local clones to make such a \"collapse\" of histories.
+"""]]

removed
diff --git a/doc/tips/redacting_history_by_converting_git_files_to_annexed/comment_1_68e96510108ca47c80c8a9ac671c5878._comment b/doc/tips/redacting_history_by_converting_git_files_to_annexed/comment_1_68e96510108ca47c80c8a9ac671c5878._comment
deleted file mode 100644
index bf3a6270ca..0000000000
--- a/doc/tips/redacting_history_by_converting_git_files_to_annexed/comment_1_68e96510108ca47c80c8a9ac671c5878._comment
+++ /dev/null
@@ -1,11 +0,0 @@
-[[!comment format=mdwn
- username="yarikoptic"
- avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
- subject="comment 1"
- date="2024-02-11T18:50:18Z"
- content="""
-Thank you!  In my case though it is pretty much the entire history of the What I had in mind is a little different and probably not really possible: I wanted to have a completely disconnected history with a commit which had `$privatefile`s but in effect you are doing the same so I think the approach is \"the same\" in effect.
-The only thing I would do differently is to first convert files to git-annex in `master` (to become `unredacted-master`), so I end up with the same tree in `unredacted-master` and `master` happen that later I would need to cherry-pick some changes accidentally committed (e.g. by collaborators) on top of `unredacted-master`.
-
-Another note worthwhile making IMHO is that AFAIK those `git replace` markers are local only, and whoever has unredacted-master later on might need to set them up as well for their local clones to make such a \"collapse\" of histories.
-"""]]

Added a comment
diff --git a/doc/tips/redacting_history_by_converting_git_files_to_annexed/comment_1_68e96510108ca47c80c8a9ac671c5878._comment b/doc/tips/redacting_history_by_converting_git_files_to_annexed/comment_1_68e96510108ca47c80c8a9ac671c5878._comment
new file mode 100644
index 0000000000..bf3a6270ca
--- /dev/null
+++ b/doc/tips/redacting_history_by_converting_git_files_to_annexed/comment_1_68e96510108ca47c80c8a9ac671c5878._comment
@@ -0,0 +1,11 @@
+[[!comment format=mdwn
+ username="yarikoptic"
+ avatar="http://cdn.libravatar.org/avatar/f11e9c84cb18d26a1748c33b48c924b4"
+ subject="comment 1"
+ date="2024-02-11T18:50:18Z"
+ content="""
+Thank you!  In my case though it is pretty much the entire history of the What I had in mind is a little different and probably not really possible: I wanted to have a completely disconnected history with a commit which had `$privatefile`s but in effect you are doing the same so I think the approach is \"the same\" in effect.
+The only thing I would do differently is to first convert files to git-annex in `master` (to become `unredacted-master`), so I end up with the same tree in `unredacted-master` and `master` happen that later I would need to cherry-pick some changes accidentally committed (e.g. by collaborators) on top of `unredacted-master`.
+
+Another note worthwhile making IMHO is that AFAIK those `git replace` markers are local only, and whoever has unredacted-master later on might need to set them up as well for their local clones to make such a \"collapse\" of histories.
+"""]]

top
diff --git a/doc/forum/move_files_under_git-annex_and_graft_history__63__/comment_5_d536cb5c2622d18802e6a32b03102f14._comment b/doc/forum/move_files_under_git-annex_and_graft_history__63__/comment_5_d536cb5c2622d18802e6a32b03102f14._comment
new file mode 100644
index 0000000000..f992f50f2b
--- /dev/null
+++ b/doc/forum/move_files_under_git-annex_and_graft_history__63__/comment_5_d536cb5c2622d18802e6a32b03102f14._comment
@@ -0,0 +1,8 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 5"""
+ date="2024-02-10T16:49:33Z"
+ content="""
+Wrote up a tip with an even more polished version of that,
+[[tips/redacting_history_by_converting_git_files_to_annexed]]
+"""]]
diff --git a/doc/tips/redacting_history_by_converting_git_files_to_annexed.mdwn b/doc/tips/redacting_history_by_converting_git_files_to_annexed.mdwn
new file mode 100644
index 0000000000..5eca035138
--- /dev/null
+++ b/doc/tips/redacting_history_by_converting_git_files_to_annexed.mdwn
@@ -0,0 +1,53 @@
+git-annex can be used to let people clone a repository, without being
+able to access the content of annexed files whose content you want to keep
+private.
+
+But what to do if you're using a repository like that, and accidentially
+add a file to git that you intended to keep private? And you didn't notice
+for a while and made other changes.
+
+Here's a way to recover from that mistake without throwing away the commits
+you made. It creates a separate, redacted history where the private
+file (`$privatefile`) is an annexed file. And uses `git replace` to let you
+locally keep using the unredacted history.
+
+Start by identifiying the parent commit of the commit that added the
+private file to git (`$lastsafecommit`).
+
+Reset back to `$lastsafecommit` and squash in all changes made since then:
+
+    git branch unredacted-master master
+    git reset --hard $lastsafecommit
+    git merge --squash unredacted-master
+
+Then convert `$privatefile` to an annexed file:
+
+    git rm --cached $privatefile
+    git annex add --force-large $privatefile
+
+Commit the redacted version of master, and locally replace it with your
+original unredacted history.
+
+    git commit
+    git replace master unredacted-master
+
+Now you can push master to other remotes, and it will push the redacted
+form of master:
+
+    git push --force origin master
+
+(Note that, if you already pushed the unredacted commits to origin, this
+push will not necessarily completely delete the private content from it.
+Making a new origin repo and pushing to that is an easy way to be sure.)
+
+If you do want to share the unredacted history with any other repositories,
+you can, by fetching the replacement refs into them:
+
+    git fetch myhost:myrepo 'refs/replace/*'
+    git fetch myhost:myrepo unredacted-master
+    git replace master unredacted-master
+
+Note that the above example makes the redacted history contain a single
+squashed commit, but this technique is not limited to that. You can make
+redacted versions of individual commits too, and build up whatever form of
+redacted history you want to publish.

improved approach
diff --git a/doc/forum/move_files_under_git-annex_and_graft_history__63__/comment_4_73e37a27ae3ff6673f34d8b1a6bed4a0._comment b/doc/forum/move_files_under_git-annex_and_graft_history__63__/comment_4_73e37a27ae3ff6673f34d8b1a6bed4a0._comment
new file mode 100644
index 0000000000..d4fb071421
--- /dev/null
+++ b/doc/forum/move_files_under_git-annex_and_graft_history__63__/comment_4_73e37a27ae3ff6673f34d8b1a6bed4a0._comment
@@ -0,0 +1,26 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 4"""
+ date="2024-02-10T16:06:26Z"
+ content="""
+A simpler approach is to make a redacted history, publish that, and locally
+replace the redacted history with the unredacted history.
+
+1. make a new empty bare repo (eg on github)
+2. git branch unredacted-master master
+3. git branch -D master
+4. start a new master branch at the commit before the problem file
+   was added, that contains one or more rewritten commits, where the problem
+   file is added to the annex (eg, use `git merge --squash
+   unredacted-master` and the convert the problem file to annexed before
+   committing)
+4. push the (new) master branch to the new bare repo
+5. `git replace master unredacted-master`
+
+That last step lets you locally access all your unredacted history locally,
+but pushes of further changes to master will still push the redacted history.
+
+You can do the same git replace in each existing clone of the repository
+and keep on referring to the unredacted history in those, while publishing
+the redacted history.
+"""]]

remove comment that doesn't actually work
diff --git a/doc/forum/move_files_under_git-annex_and_graft_history__63__/comment_4_f07514643cbc841f79135ed5db9fe022._comment b/doc/forum/move_files_under_git-annex_and_graft_history__63__/comment_4_f07514643cbc841f79135ed5db9fe022._comment
deleted file mode 100644
index c32e36ca27..0000000000
--- a/doc/forum/move_files_under_git-annex_and_graft_history__63__/comment_4_f07514643cbc841f79135ed5db9fe022._comment
+++ /dev/null
@@ -1,48 +0,0 @@
-[[!comment format=mdwn
- username="joey"
- subject="""comment 4"""
- date="2024-02-10T14:50:03Z"
- content="""
-Another way to use git replace, that is simpler, and requires less
-repository hacking:
-
-Checkout the commit just before that problem file was added:
-
-	git checkout 32358
-
-Squash merge the content of master into the working tree:
-
-	git merge --squash master
-
-Convert the problem file to an annexed file:
-
-	git rm --cached $file
-	git annex add --force-large $file
-
-Commit the result and replace the master branch with the new commit:
-
-	git commit -m 'rewritten history with $file converted to annexed'
-	git replace master HEAD
-
-Now you can checkout the master branch, and see the file is annexed there
-now:
-
-	git checkout master
-	git-annex find $file
-
-And finally, make one more commit, which can be empty, to have something
-new to force push to origin:
-
-	git commit --allow-empty -m 'empty commit after rewrite of history'
-	git push --force origin master
-
-Now a new clone from origin will get only the rewritten history, and there
-is no need to mess with replace refs, and no errors. You will still have
-access to the original history in your repo, and can interoperate with
-those clones. And origin will still contain the content of the problem file
-until you go in and delete it.
-
-Of course, it's also possible, rather than squashing the whole problematic
-history into one commit, to regenerate specific commits to use the annexed
-file.
-"""]]

update
diff --git a/doc/todo/verified_relaxed_urls.mdwn b/doc/todo/verified_relaxed_urls.mdwn
index a5aa18cce8..90e76cd09c 100644
--- a/doc/todo/verified_relaxed_urls.mdwn
+++ b/doc/todo/verified_relaxed_urls.mdwn
@@ -28,7 +28,8 @@ download from the web. (Call this a "dynamic" url key.)
 And handle all existing relaxed url keys as before.
 
 That would leave it up to the user to migrate their relaxed url keys to
-dynamic urls keys, if desired.
+dynamic urls keys, if desired. Now that distributed migration is
+implemented, that seems sufficiently easy.
 
 ## other special remotes
 
@@ -62,3 +63,5 @@ compared with migrating the key to the desired hash backend.
 
 Does seem to be some chance this could help implementing 
 [[wishlist_degraded_files]].
+
+[[!tag projects/datalad/potential]]

comment
diff --git a/doc/forum/move_files_under_git-annex_and_graft_history__63__/comment_4_f07514643cbc841f79135ed5db9fe022._comment b/doc/forum/move_files_under_git-annex_and_graft_history__63__/comment_4_f07514643cbc841f79135ed5db9fe022._comment
new file mode 100644
index 0000000000..c32e36ca27
--- /dev/null
+++ b/doc/forum/move_files_under_git-annex_and_graft_history__63__/comment_4_f07514643cbc841f79135ed5db9fe022._comment
@@ -0,0 +1,48 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 4"""
+ date="2024-02-10T14:50:03Z"
+ content="""
+Another way to use git replace, that is simpler, and requires less
+repository hacking:
+
+Checkout the commit just before that problem file was added:
+
+	git checkout 32358
+
+Squash merge the content of master into the working tree:
+
+	git merge --squash master
+
+Convert the problem file to an annexed file:
+
+	git rm --cached $file
+	git annex add --force-large $file
+
+Commit the result and replace the master branch with the new commit:
+
+	git commit -m 'rewritten history with $file converted to annexed'
+	git replace master HEAD
+
+Now you can checkout the master branch, and see the file is annexed there
+now:
+
+	git checkout master
+	git-annex find $file
+
+And finally, make one more commit, which can be empty, to have something
+new to force push to origin:
+
+	git commit --allow-empty -m 'empty commit after rewrite of history'
+	git push --force origin master
+
+Now a new clone from origin will get only the rewritten history, and there
+is no need to mess with replace refs, and no errors. You will still have
+access to the original history in your repo, and can interoperate with
+those clones. And origin will still contain the content of the problem file
+until you go in and delete it.
+
+Of course, it's also possible, rather than squashing the whole problematic
+history into one commit, to regenerate specific commits to use the annexed
+file.
+"""]]