Please describe the problem.
Doing a HEAD request against the /git-annex/$uuid/key/$key endpoint and also any of the versioned key endpoints works the first time, but after that the p2phttp process stops answering any requests.
This is independent of the values for --jobs and --cpus, so does not seem to be another case of blocked worker processes.
What steps will reproduce the problem?
- Serve a repository with p2phttp:
git -C . annex p2phttp --jobs 200 --cpus 4 --bind 127.0.0.1 --wideopen --port 54321 --debug - Do a HEAD request:
curl --head http://localhost:54321/git-annex/df5642e3-3777-42ac-8c40-f9c1c7264e66/key/MD5E-s5--2b00042f7481c7b056c4b410d28f33cf(substitute uuid and key for valid values) - Do the same HEAD request again, observe it getting stuck
What version of git-annex are you using? On what operating system?
git-annex version: 10.20260421-gd789e57311eca6bbf4949a6f22b44dd46aabb9b1
build flags: Assistant Webapp Inotify TorrentParser MagicMime Benchmark Feeds Testsuite S3 WebDAV Servant OsPath
dependency versions: aws-0.25.2 bloomfilter-2.0.1.3 crypton-1.0.4 DAV-1.3.4 feed-1.3.2.1 ghc-9.10.3 http-client-0.7.19 torrent-10000.1.3 uuid-1.3.16 yesod-1.6.2.1
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL GITBUNDLE GITMANIFEST VURL X*
remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs httpalso borg rclone hook external compute mask
operating system: linux x86_64
supported repository versions: 8 9 10
upgrade supported from repository versions: 0 1 2 3 4 5 6 7 8 9 10
local repository version: 10
Please provide any additional information below.
p2phttp output:
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log
$ git -C . annex p2phttp --jobs 200 --cpus 4 --bind 127.0.0.1 --wideopen --port 54321 --debug
[2026-05-15 11:20:23.895756881] (Utility.Process) process [652639] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","show-ref","git-annex"]
[2026-05-15 11:20:23.896781487] (Utility.Process) process [652639] done ExitSuccess
[2026-05-15 11:20:23.896998219] (Utility.Process) process [652640] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","show-ref","--hash","refs/heads/git-annex"]
[2026-05-15 11:20:23.897894537] (Utility.Process) process [652640] done ExitSuccess
[2026-05-15 11:20:23.89863384] (Utility.Process) process [652641] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","cat-file","--batch"]
[2026-05-15 11:20:23.900090498] (Utility.Process) process [652642] read: git ["-c","safe.directory=*","-c","safe.bareRepository=all","config","--null","--list"] in "./../a1-bare"
[2026-05-15 11:20:23.901180123] (Git.Config) git config read: [("",[""]),("alias.gone",["! git fetch -p && git for-each-ref --format '%(refname:short) %(upstream:track)' | awk '$2 == \"[gone]\" {print $1}' | xargs -r git branch -D"]),("alias.lol",["log --graph --decorate --pretty=oneline --abbrev-commit"]),("alias.lola",["log --graph --decorate --pretty=oneline --abbrev-commit --all"]),("alias.wdiff",["diff --color-words=."]),("annex.url",["annex+http://localhost:54321"]),("annex.uuid",["184144b2-f69b-46a2-80e2-58bfb149e11d"]),("annex.version",["10"]),("branch.sort",["-committerdate"]),("color.branch",["auto"]),("color.diff",["auto"]),("color.interactive",["auto"]),("color.status",["auto"]),("column.ui",["auto"]),("commit.verbose",["true"]),("core.bare",["true"]),("core.filemode",["true"]),("core.repositoryformatversion",["0"]),("credential.helper",["oauth -verbose","cache --timeout 21600"]),("credential.https://jugit.fz-juelich.de.oauthclientid",["9d539aa7c067a768bfea62e27daf6da2cd8c0242063f068508222404b94896e2"]),("datalad.create-sibling-ghlike.extra-remote-settings.atris.fz-juelich.de.annex-ignore",["false"]),("datalad.credential.demo.dataverse.org.last-used",["2025-02-27T10:03:48.565458"]),("datalad.credential.demo.dataverse.org.realm",["https://demo.dataverse.org/dataverse"]),("datalad.credential.demo.dataverse.org.type",["token"]),("diff.annextextdiff.command",["git annex diffdriver --text"]),("diff.colormoved",["true"]),("diff.wordregex",["."]),("filter.lfs.clean",["git-lfs clean -- %f"]),("filter.lfs.process",["git-lfs filter-process"]),("filter.lfs.required",["true"]),("filter.lfs.smudge",["git-lfs smudge -- %f"]),("http.sslcainfo",["/home/icg149/.pixi/envs/git/ssl/cacert.pem"]),("http.sslcapath",["/home/icg149/.pixi/envs/git/ssl/cacert.pem"]),("http.sslverify",["true"]),("init.defaultbranch",["main"]),("merge.conflictstyle",["diff3"]),("merge.mergiraf.driver",["mergiraf merge --git %O %A %B -s %S -x %X -y %Y -p %P -l %L"]),("merge.mergiraf.name",["mergiraf"]),("rebase.autosquash",["true"]),("rebase.autostash",["true"]),("rebase.updaterefs",["true"]),("rerere.autoupdate",["true"]),("rerere.enabled",["true"]),("safe.barerepository",["all"]),("safe.directory",["*"]),("tag.sort",["version:refname"]),("user.email",["m.risse@fz-juelich.de"]),("user.name",["Matthias Ri\223e"])]
[2026-05-15 11:20:23.901462068] (Utility.Process) process [652642] done ExitSuccess
[2026-05-15 11:20:23.901933159] (Annex.Branch) read proxy.log
[2026-05-15 11:20:37.787308378] (P2P.IO) [http client] [ThreadId 20] P2P > GET 0 MD5E-s5--2b00042f7481c7b056c4b410d28f33cf
[2026-05-15 11:20:37.787517924] (P2P.IO) [http server] [ThreadId 19] P2P < GET 0 MD5E-s5--2b00042f7481c7b056c4b410d28f33cf
[2026-05-15 11:20:37.788184946] (Utility.Process) process [652703] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","-c","filter.annex.smudge=","-c","filter.annex.clean=","-c","filter.annex.process=","write-tree"]
[2026-05-15 11:20:37.790260868] (Utility.Process) process [652703] done ExitSuccess
[2026-05-15 11:20:37.790588597] (Utility.Process) process [652704] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","show-ref","--hash","refs/annex/last-index"]
[2026-05-15 11:20:37.791806257] (Utility.Process) process [652704] done ExitSuccess
[2026-05-15 11:20:37.792481433] (P2P.IO) [http server] [ThreadId 19] P2P > DATA 5
[2026-05-15 11:20:37.792564378] (P2P.IO) [http client] [ThreadId 20] P2P < DATA 5
# End of transcript or log.
curl requests:
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log
$ curl --head http://localhost:54321/git-annex/df5642e3-3777-42ac-8c40-f9c1c7264e66/key/MD5E-s5--2b00042f7481c7b056c4b410d28f33cf
HTTP/1.1 200 OK
Date: Fri, 15 May 2026 09:20:37 GMT
Server: Warp/3.4.9
Content-Type: application/octet-stream
X-git-annex-data-length: 5
icg149@icg1911:~/Playground/c1$ curl --head http://localhost:54321/git-annex/df5642e3-3777-42ac-8c40-f9c1c7264e66/key/MD5E-s5--2b00042f7481c7b056c4b410d28f33cf
[stuck]
# End of transcript or log.
Reproduced this.
I'm surprised it responds to HEAD at all. It's not a documented part of the p2phttp API, and the implementation is only a GET endpoint. I guess that
servantmakes GET endpoints also support HEAD? Urk.It seems possible that this doesn't only happen on HEAD, but also on a GET where the client disconnects without reading any of the response body. The code path through looks like it would possibly be the same.
It is getting stuck on
getP2PConnection. So far I've determined that the connection servicer thread gets stuck handling a connection release. Which is why the subsequent HEAD fails. So will any subsequent request actually. So this can take down a p2phttp server with a single request.Fixed this.
(I do think it could have also happened without HEAD with just the right timing of the client hanging up on GET, still have not verified that. Of course, we had a whole bug about p2phttp can get stuck with interrupted clients that was dealt with previously, but maybe we missed it back then.)
I've also documented
HEAD /git-annex/$uuid/key/$keyas supported by p2phttp because if you give a HTTP client an URL, I suppose it may try HEAD.I would rather that the versioned GET endpoints not also support HEAD, just because it's not part of the interface git-annex uses. If I find a way to prevent
servantfrom automatically supporting HEAD for those, I will use it.Yes, I think all of the "higher-level http server frameworks" I've encountered (definitely Flask and the construct Forgejo is using, but also others) automatically support HEAD for all GET endpoints, because a properly implemented HEAD is a subset of GET anyway. I'd expect servant to do the same.
At least I didn't get the p2phttp server stuck with interrupted clients while investigating this issue (that was my initial guess on what was causing the server to get stuck in the first place), but I did see a different bug that I didn't yet report which caused the p2phttp server to exit with exit code 141 if a client was interrupted at the "right" time. This one might already be fixed by https://git-annex.branchable.com/bugs/SIGPIPE_behavior_change/ though.
The initial use-case by mih was to point
git annex addurlat this key endpoint, and that does try HEAD, which triggered the bug. So even git-annex itself does it, it just fell out of the report when I reduced the reproducer as far as possibleThank you!