Another approach would be to keep a bounded pool of PerRepoServerStates. When a PerRepoServerState needs to be added to the pool and the pool is full, first find one for a different repository that is not in use, and remove it from the pool, stopping its helper processes.

This seems easier to implement than coordinating a process pool between Annex objects.

And it could also avoid problems with too many open files, since it could also close open file handles in the Annex object such as the keys db handle. I do think that running out of file descriptors would be the next problem after too many processes when scaling p2phttp up to serving a large number of repositories, so this might as well solve that too.

There would still be worker pools for processes like git hash-object that are shared amoung all threads serving the same repository, scaling up to one per cpu. So long as the pool is capped at some reasonable number (say 100), that is ok, it means something like 100 * (cpus + 1) processes, probably well under the maximum number of processes. Make the maximum concurrent repositories served a tunable value.

I think this is the problem as the (already fixed) info fails to resolve from submodule in worktree.

This error message is displayed when S3 getObject fails with a 412 "Precondition Failed".

I suspect you may be running into S3 eventual consistency issues.

The key is of course the one pointed to by the file that it's acting on when this error is displayed. So I don't see any reason for git-annex to display the key.

As for displaying the versionid or etag, I think that would already be done by --debug when it shows the http request headers.

I did develop this patch to display that as part of the error message, but I'd only be inclined to apply it if --debug doesn't have the information.

diff --git a/Remote/S3.hs b/Remote/S3.hs
index bf77e56419..74569a0a3a 100644
--- a/Remote/S3.hs
+++ b/Remote/S3.hs
@@ -762,9 +762,10 @@ retrieveExportWithContentIdentifierS3 hv r rs info loc (cid:_) dest gk p =
   where
    go iv = withS3Handle hv $ \case
        Right h -> do
-           rewritePreconditionException $ retrieveHelper' (gitconfig r) info h dest p iv $
-               limitGetToContentIdentifier cid $
-                   S3.getObject (bucket info) o
+           rewritePreconditionException cid o $
+               retrieveHelper' (gitconfig r) info h dest p iv $
+                   limitGetToContentIdentifier cid $
+                       S3.getObject (bucket info) o
            k <- either return id gk
            case extractContentIdentifier cid o of
                Right vid -> do
@@ -779,12 +780,16 @@ retrieveExportWithContentIdentifierS3 _ _ _ _ _ [] _ _ _ = giveup "missing conte

 {- Catch exception getObject returns when a precondition is not met,
  - and replace with a more understandable message for the user. -}
-rewritePreconditionException :: Annex a -> Annex a
-rewritePreconditionException a = catchJust (Url.matchStatusCodeException want) a $ 
-   const $ giveup "requested version of object is not available in S3 bucket"
+rewritePreconditionException :: ContentIdentifier -> S3.Object -> Annex a -> Annex a
+rewritePreconditionException cid o a = catchJust (Url.matchStatusCodeException want) a $ 
+   const $ giveup $ "requested version of object (" ++ ciddesc ++ ") ++ is not available in S3 bucket"
   where
    want st = statusCode st == 412 && 
        statusMessage st == "Precondition Failed"
+   ciddesc = case extractContentIdentifier cid o of
+       Left etag -> "etag " ++ T.unpack etag
+       Right (Just (S3VersionID _ vid)) -> T.unpack vid
+       Right Nothing -> mempty

 -- Does not check if content on S3 is safe to overwrite, because there
 -- is no atomic way to do so. When the bucket is versioned, this is

Thanks for the pointer to the CHANGELOG. I'd missed that file once I'd cloned the repo. FWIW the CHANGELOG (a) doesn't seem to be linked anywhere obvious for web viewing, (b) the gitweb view is pretty hidden (only linked on the download page, which is possibly only linked from the install/fromsource page), and (c) the gitweb is quite slow. So at present cloning the git repo is probably the easiest way to to read the CHANGELOG (versus doc/news which is on the web site, but only a recent subset).

Possibly there's some way to make the CHANGELOG file available as a plain text view on the web more directly than "find the gitweb, navigate to the tree"? And maybe link it from a doc/news article that sorts just after the "recent changes", as a "for older release notes, see...".

For the record, from some digging my guess is that the macOS 10.15 to macOS 14/15 minimum version change happened when the builds moved to GitHub Actions (and thus GitHub Runners), which as best I can tell was around May 2026. So builds up to April 2026 might also work on macOS 10.15. But I didn't see anything particularly relevant to my use case between January 2026 and April 2026, so I've not tried any builds newer than December 2025 on the macOS 10.15 system.

I'd tend to agree with just having a single macOS build at this point. The next likely incompatible change is to stop building Intel/macOS "fat" binaries, which I suspect some combination of Apple and/or GitHub will force on you sometime in the next 12 months. But that's a very obvious failure case -- you get "Bad CPU type in executable" when trying to run anything if running an ARM-only build on an Intel system.

So it'd be worth documenting the expected minimum supported macOS for the standalone build (here macOS 15, or maybe macOS 14), but it doesn't need to be in the directory name. And it'd be worth noting when the builds go ARM/M1-only (which I'd guess will be when GitHub remove the macOS Intel runners, likely in 2027, or when macOS 15 goes unsupported).

If you haven't already, I think you can mark this issue as "done" at your convenience, as I have setup that seems to work now, and the directory name is no longer misleading.

Ewen

bisected to f0c046f016aab6485eea57101df953895728e8b4, "re-probing of annex.url for annex+https on connection failure".

The CHANGELOG file is what you want I think.

I've updated the website re MacPorts.

I suspect it may even need macOS 15. That's the OS that the build is happening on. I don't think it makes sense to include the supported OS version in the directory name really; it's unlikely there would be 2 incompatible versions that both would have builds happening.

Also in case it helps someone else later, as best I can tell the release notes are in the git repository as doc/news/version_10.20xxxxxx.mdwn, but it seems like the older release notes get deleted when newer ones are added (at a guess, to keep ~4 release notes showing up on the news page).

This makes reviewing the release notes since an older release somewhat more difficult than it might be, as it seems like the only record of the older release notes is as deleted files, in the git history.

I ended up coming up with:

git clone git://git-annex.branchable.com/ git-annex
cd git-annex/doc/news
git log --stat f1985eabce137adb8851cbbaea1fa8474406f7b5..HEAD . | egrep '^commit|10.2026.*\+\+\+\+\+\+' | grep -B 1 doc/news | grep -v '^--' | sed 'N;s/\n/ /; s/^commit/git show/; s/  doc/:doc/; s/mdwn.*$/mdwn/' | tac | sh -x

to see the release notes for 2026, in chronologically increasing order (ie oldest to newest). Where the magic git commit hash was found by searching with "git log" for the last 2025 / first 2026 commit to doc/news, since the version I found that ran for me (on macOS 10.15) was the last 2025 standalone build, and thus I wanted to know what had changed since then.

(Possibly older news articles could be git mv-ed to an archive directory rather than git rm-ed to make this easier in future, without cluttering up the news page?)

Anyway skim reading the least year's worth of release notes (from the above command, and some slight tidying up), I don't see much that an internal-only git annex remote-only would particularly need before the various October 2026 fixes (which I can't easily have due to build dependencies not supporting macOS 10.15). The only thing that stood out as potentially desirable is the hash performance improvements in git-annex 10.20260624, but for this mostly-archivable, infrequently updated use case I can probably live without those too.

Ewen