I'm surprised it responds to HEAD at all. It's not a documented part of the p2phttp API, and the implementation is only a GET endpoint. I guess that servant makes GET endpoints also support HEAD? Urk.

Yes, I think all of the "higher-level http server frameworks" I've encountered (definitely Flask and the construct Forgejo is using, but also others) automatically support HEAD for all GET endpoints, because a properly implemented HEAD is a subset of GET anyway. I'd expect servant to do the same.

(I do think it could have also happened without HEAD with just the right timing of the client hanging up on GET, still have not verified that. Of course, we had a whole bug about p2phttp can get stuck with interrupted clients that was dealt with previously, but maybe we missed it back then.)

At least I didn't get the p2phttp server stuck with interrupted clients while investigating this issue (that was my initial guess on what was causing the server to get stuck in the first place), but I did see a different bug that I didn't yet report which caused the p2phttp server to exit with exit code 141 if a client was interrupted at the "right" time. This one might already be fixed by https://git-annex.branchable.com/bugs/SIGPIPE_behavior_change/ though.

I've also documented HEAD /git-annex/$uuid/key/$key as supported by p2phttp because if you give a HTTP client an URL, I suppose it may try HEAD.

The initial use-case by mih was to point git annex addurl at this key endpoint, and that does try HEAD, which triggered the bug. So even git-annex itself does it, it just fell out of the report when I reduced the reproducer as far as possible

Fixed this.

Thank you!

Fixed this.

(I do think it could have also happened without HEAD with just the right timing of the client hanging up on GET, still have not verified that. Of course, we had a whole bug about p2phttp can get stuck with interrupted clients that was dealt with previously, but maybe we missed it back then.)

I've also documented HEAD /git-annex/$uuid/key/$key as supported by p2phttp because if you give a HTTP client an URL, I suppose it may try HEAD.

I would rather that the versioned GET endpoints not also support HEAD, just because it's not part of the interface git-annex uses. If I find a way to prevent servant from automatically supporting HEAD for those, I will use it.

Reproduced this.

I'm surprised it responds to HEAD at all. It's not a documented part of the p2phttp API, and the implementation is only a GET endpoint. I guess that servant makes GET endpoints also support HEAD? Urk.

It seems possible that this doesn't only happen on HEAD, but also on a GET where the client disconnects without reading any of the response body. The code path through looks like it would possibly be the same.

It is getting stuck on getP2PConnection. So far I've determined that the connection servicer thread gets stuck handling a connection release. Which is why the subsequent HEAD fails. So will any subsequent request actually. So this can take down a p2phttp server with a single request.

Note that the transcripts are not quite what git-annex usually outputs, due to this bug, which has now been fixed.

I have long disliked how this is displayed in the ssh case too.

At the level of the P2P protocol, a solution to this could be for the server to send an ERROR message back while the client is still in the process of sending the file with DATA. The P2P protocol allows ERROR to be sent at any time.

This would look something like P2P.IO.runNet in SendBytes on error, trying to getProtocolLine, and when it gets an ERROR returning the error message as the Left ProtoFailureMessage rather than the current exception.

(Something would need to be done in the P2PHandleTMVar to handle proxying too.)

For the P2P protocol over http, the /put response would look something like:

{"stored": false, "error-message": "not enough free space, need 1.05 GB more}"}

Currently p2phttp actually replies with 500 Internal Server Error, which git-annex does display to the user.

The other side of the problem though is that the disk space message is displayed as a warning. So how would git-annex-shell, or p2phttp intercept it to send it along to the client? There would need to be quite a lot of restructuring to make that an exception.

There are other warnings as well that it would be good to send to the client. One that comes to mind is "transfer already in progress, or unable to take transfer lock". So this is a more general problem.

I wanted to investigate further and added a credential "helper" that documents what was queried

cat << EOT > /usr/local/bin/git-credential-echo
#!/usr/bin/env bash
exec cat >&2
EOT
chmod +x /usr/local/bin/git-credential-echo
git config --global --add credential.helper echo

I also switched from annex push to annex copy (because this is the aspect that failed). I now see (what I could have seen in the log above already). The issue is not that the credential isn't retrieved properly. It is actually rejected, and the superficial/original error is the result of prompting for another valid credential. Here is the log of a copy call:

git annex --debug copy -t origin .
[2026-05-16 10:01:43.073507233] (Utility.Process) process [639] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","show-ref","git-annex"]
[2026-05-16 10:01:43.075431464] (Utility.Process) process [639] done ExitSuccess
[2026-05-16 10:01:43.075932187] (Utility.Process) process [640] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","show-ref","--hash","refs/heads/git-annex"]
[2026-05-16 10:01:43.077917969] (Utility.Process) process [640] done ExitSuccess
[2026-05-16 10:01:43.078259638] (Annex.Branch) read remote.log
[2026-05-16 10:01:43.079214293] (Utility.Process) process [641] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","cat-file","--batch"]
[2026-05-16 10:01:43.081261767] (Annex.Branch) read proxy.log
[2026-05-16 10:01:43.082419578] (Utility.Process) process [642] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","ls-files","--stage","-z","--error-unmatch","--","."]
[2026-05-16 10:01:43.082798858] (Utility.Process) process [643] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","cat-file","--batch-check=%(objectname) %(objecttype) %(objectsize)","--buffer"]
[2026-05-16 10:01:43.083296281] (Utility.Process) process [644] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","cat-file","--batch=%(objectname) %(objecttype) %(objectsize)","--buffer"]
[2026-05-16 10:01:43.083778403] (Utility.Process) process [641] done ExitSuccess
[2026-05-16 10:01:43.086359241] (Utility.Process) process [645] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","cat-file","--batch=%(objectname) %(objecttype) %(objectsize)","--buffer"]
copy static/graph.json (to origin...) [2026-05-16 10:01:43.210382475] (Utility.Process) process [647] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","-c","filter.annex.smudge=","-c","filter.annex.clean=","-c","filter.annex.process=","write-tree"]
[2026-05-16 10:01:43.214071033] (Utility.Process) process [647] done ExitSuccess
[2026-05-16 10:01:43.214666158] (Utility.Process) process [648] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","show-ref","--hash","refs/annex/last-index"]
[2026-05-16 10:01:43.218272853] (Utility.Process) process [648] done ExitSuccess
[2026-05-16 10:01:43.218310804] (Database.Keys) reconcileStaged start
[2026-05-16 10:01:43.218806637] (Utility.Process) process [649] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","cat-file","--batch-check=%(objectname) %(objecttype) %(objectsize)","--buffer"]
[2026-05-16 10:01:43.219327951] (Utility.Process) process [650] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","cat-file","--batch=%(objectname) %(objecttype) %(objectsize)","--buffer"]
[2026-05-16 10:01:43.219921627] (Utility.Process) process [651] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","-c","filter.annex.smudge=","-c","filter.annex.clean=","-c","filter.annex.process=","-c","diff.external=","diff","ee47aafecb91c163b0eb9e7ef1a35b07d5b1e0b9","8ea4ce9a4065396e07306bc2f30bcf295837ad6f","--raw","-z","--no-abbrev","-G/annex/objects/","--no-renames","--ignore-submodules=all","--no-textconv","--no-ext-diff"]
[2026-05-16 10:01:43.223295855] (Utility.Process) process [651] done ExitSuccess
[2026-05-16 10:01:43.225251807] (Database.Handle) commitDb start
[2026-05-16 10:01:43.225610276] (Database.Handle) commitDb done
[2026-05-16 10:01:43.225676608] (Utility.Process) process [650] done ExitSuccess
[2026-05-16 10:01:43.2257297] (Utility.Process) process [649] done ExitSuccess
[2026-05-16 10:01:43.226178161] (Utility.Process) process [652] call: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","update-ref","refs/annex/last-index","8ea4ce9a4065396e07306bc2f30bcf295837ad6f"]
[2026-05-16 10:01:43.228765129] (Utility.Process) process [652] done ExitSuccess
[2026-05-16 10:01:43.22880699] (Database.Keys) reconcileStaged end
[2026-05-16 10:01:43.246406143] (Utility.Process) process [653] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","credential","fill"]
[2026-05-16 10:01:43.253477499] (Utility.Process) process [653] done ExitSuccess
[2026-05-16 10:01:43.274667127] (Utility.Process) process [656] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","credential","reject"]
protocol=https
host=hub.psychoinformatics.de
username=myuser
password=***
[2026-05-16 10:01:43.2873155] (Utility.Process) process [656] done ExitSuccess
25%   31.98 KiB        70 MiB/s 0s
100%  126.9 KiB       179 MiB/s 0s[2026-05-16 10:01:43.330200019] (Utility.Process) process [662] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","credential","fill"]
protocol=https
host=hub.psychoinformatics.de
fatal: could not read Username for 'https://hub.psychoinformatics.de': No such device or address
[2026-05-16 10:01:43.341958838] (Utility.Process) process [662] done ExitFailure 128

  user error (git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","credential","fill"] exited 128)
[2026-05-16 10:01:43.357710043] (Utility.Process) process [668] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","credential","fill"]
protocol=https
host=hub.psychoinformatics.de
fatal: could not read Username for 'https://hub.psychoinformatics.de': No such device or address
[2026-05-16 10:01:43.369272597] (Utility.Process) process [668] done ExitFailure 128

  user error (git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","credential","fill"] exited 128)

failed
[2026-05-16 10:01:43.371334491] (Utility.Process) process [645] done ExitSuccess
[2026-05-16 10:01:43.371445144] (Utility.Process) process [644] done ExitSuccess
[2026-05-16 10:01:43.371533016] (Utility.Process) process [643] done ExitSuccess
[2026-05-16 10:01:43.371599238] (Utility.Process) process [642] done ExitSuccess
copy: 1 failed

I am now seeing the "same" problem in a different context, where no 3rd-party credential helper is involved.

Concretely, I have a Forgejo actions workflow that manually pre-fills a credential via

git config --global credential.helper cache
git credential approve <<EOF
protocol=https
host=hub.psychoinformatics.de
username=${{ forgejo.actor }}
password=${{ forgejo.token }}
EOF

This appears to be working, as confirmed by

git credential fill
protocol=https
host=hub.psychoinformatics.de

which returns

username=myuser
password=***

However, running git annex push immediately afterwards has the git-push parts working, but the internal credential fill fail with fatal: could not read Username for 'https://hub.psychoinformatics.de': No such device or address (full log below).

+ git annex --debug push
[2026-05-16 09:19:30.185331705] (Utility.Process) process [597] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","show-ref","git-annex"]
[2026-05-16 09:19:30.189155934] (Utility.Process) process [597] done ExitSuccess
[2026-05-16 09:19:30.189885413] (Utility.Process) process [598] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","show-ref","--hash","refs/heads/git-annex"]
[2026-05-16 09:19:30.192024278] (Utility.Process) process [598] done ExitSuccess
[2026-05-16 09:19:30.192633434] (Annex.Branch) read config.log
[2026-05-16 09:19:30.193750572] (Utility.Process) process [599] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","cat-file","--batch"]
[2026-05-16 09:19:30.195842697] (Utility.Process) process [600] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","symbolic-ref","-q","HEAD"]
[2026-05-16 09:19:30.197578872] (Utility.Process) process [600] done ExitSuccess
[2026-05-16 09:19:30.198023783] (Utility.Process) process [601] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","show-ref","refs/heads/main"]
[2026-05-16 09:19:30.19984727] (Utility.Process) process [601] done ExitSuccess
[2026-05-16 09:19:30.200553359] (Utility.Process) process [602] call: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","show-ref","--verify","-q","refs/heads/synced/main"]
[2026-05-16 09:19:30.20216315] (Utility.Process) process [602] done ExitFailure 1
[2026-05-16 09:19:30.202230582] (Annex.Branch) read remote.log
[2026-05-16 09:19:30.202912609] (Annex.Branch) read proxy.log
[2026-05-16 09:19:30.20410367] (Utility.Process) process [603] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","show-ref","git-annex"]
[2026-05-16 09:19:30.20603299] (Utility.Process) process [603] done ExitSuccess
[2026-05-16 09:19:30.206394769] (Utility.Process) process [604] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","show-ref","--hash","refs/heads/git-annex"]
[2026-05-16 09:19:30.208407081] (Utility.Process) process [604] done ExitSuccess
[2026-05-16 09:19:30.209282774] (Utility.Process) process [605] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","show-ref","--hash","refs/heads/git-annex"]
[2026-05-16 09:19:30.211913422] (Utility.Process) process [605] done ExitSuccess
[2026-05-16 09:19:30.212857467] (Utility.Process) process [606] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","log","-z","--pretty=format:%H %ct","--raw","--no-abbrev","--no-renames","--reverse","--follow","668c5aa16d4e9ee0af8fc63f4fbb3e7cfb4b93d4","--","migrate.tree"]
[2026-05-16 09:19:30.215528925] (Utility.Process) process [606] done ExitSuccess
[2026-05-16 09:19:30.216522001] (Utility.Process) process [607] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","ls-files","--stage","-z","--error-unmatch","--"]
[2026-05-16 09:19:30.217095856] (Utility.Process) process [608] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","cat-file","--batch-check=%(objectname) %(objecttype) %(objectsize)","--buffer"]
[2026-05-16 09:19:30.217808254] (Utility.Process) process [609] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","cat-file","--batch=%(objectname) %(objecttype) %(objectsize)","--buffer"]
[2026-05-16 09:19:30.218346408] (Utility.Process) process [599] done ExitSuccess
[2026-05-16 09:19:30.222153637] (Utility.Process) process [610] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","cat-file","--batch=%(objectname) %(objecttype) %(objectsize)","--buffer"]
[2026-05-16 09:19:30.228726336] (Annex.Branch) read trust.log
[2026-05-16 09:19:30.229328452] (Utility.Process) process [611] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","cat-file","--batch"]
[2026-05-16 09:19:30.230981234] (Annex.Branch) read cluster.log
[2026-05-16 09:19:30.237149214] (Utility.Process) process [614] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","-c","filter.annex.smudge=","-c","filter.annex.clean=","-c","filter.annex.process=","write-tree"]
[2026-05-16 09:19:30.239369771] (Utility.Process) process [614] done ExitSuccess
[2026-05-16 09:19:30.239845993] (Utility.Process) process [615] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","show-ref","--hash","refs/annex/last-index"]
[2026-05-16 09:19:30.241848385] (Utility.Process) process [615] done ExitSuccess
[2026-05-16 09:19:30.242685967] (Utility.Process) process [616] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","check-attr","-z","--stdin","annex.backend","annex.largefiles","annex.numcopies","annex.mincopies","--"]
[2026-05-16 09:19:30.244064203] (Annex.Branch) read numcopies.log
[2026-05-16 09:19:30.244322359] (Annex.Branch) read mincopies.log
[2026-05-16 09:19:30.257332835] (Annex.Branch) read group.log
[2026-05-16 09:19:30.25750822] (Annex.Branch) read group-preferred-content.log
[2026-05-16 09:19:30.257716505] (Annex.Branch) read preferred-content.log
[2026-05-16 09:19:30.257939851] (Annex.Branch) read required-content.log
[2026-05-16 09:19:30.258732681] (Annex.Branch) read 455/e88/MD5E-s129943--06caaa82dc6cfdd358085974adbbe8d3.json.log
copy static/graph.json (to origin...) [2026-05-16 09:19:30.378214958] (Utility.Process) process [617] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","credential","fill"]
[2026-05-16 09:19:30.383551986] (Utility.Process) process [617] done ExitSuccess
[2026-05-16 09:19:30.404495517] (Utility.Process) process [620] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","credential","reject"]
[2026-05-16 09:19:30.40926853] (Utility.Process) process [620] done ExitSuccess
25%   31.98 KiB        68 MiB/s 0s
100%  126.9 KiB       177 MiB/s 0s[2026-05-16 09:19:30.450818263] (Utility.Process) process [623] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","credential","fill"]
fatal: could not read Username for 'https://hub.psychoinformatics.de': No such device or address
[2026-05-16 09:19:30.455457733] (Utility.Process) process [623] done ExitFailure 128

  user error (git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","credential","fill"] exited 128)
[2026-05-16 09:19:30.473680804] (Utility.Process) process [626] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","credential","fill"]
fatal: could not read Username for 'https://hub.psychoinformatics.de': No such device or address
[2026-05-16 09:19:30.47933976] (Utility.Process) process [626] done ExitFailure 128

  user error (git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","credential","fill"] exited 128)

failed
[2026-05-16 09:19:30.481071965] (Utility.Process) process [610] done ExitSuccess
[2026-05-16 09:19:30.481128386] (Utility.Process) process [609] done ExitSuccess
[2026-05-16 09:19:30.481169867] (Utility.Process) process [608] done ExitSuccess
[2026-05-16 09:19:30.481201948] (Utility.Process) process [607] done ExitSuccess
[2026-05-16 09:19:30.481629439] (Utility.Process) process [629] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","symbolic-ref","-q","HEAD"]
[2026-05-16 09:19:30.483084607] (Utility.Process) process [629] done ExitSuccess
[2026-05-16 09:19:30.483620051] (Utility.Process) process [630] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","show-ref","refs/heads/main"]
[2026-05-16 09:19:30.486919556] (Utility.Process) process [630] done ExitSuccess
[2026-05-16 09:19:30.487433589] (Utility.Process) process [631] call: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","branch","-f","synced/main","refs/heads/main"]
[2026-05-16 09:19:30.490124219] (Utility.Process) process [631] done ExitSuccess
[2026-05-16 09:19:30.490606621] (Utility.Process) process [632] call: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","show-ref","--verify","-q","refs/remotes/origin/synced/main"]
[2026-05-16 09:19:30.493191278] (Utility.Process) process [632] done ExitFailure 1

push origin 

[2026-05-16 09:19:30.493862885] (Utility.Process) process [633] call: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","push","origin","main"]
Everything up-to-date
[2026-05-16 09:19:30.622077877] (Utility.Process) process [633] done ExitSuccess
[2026-05-16 09:19:30.622889428] (Utility.Process) process [637] call: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","push","origin","main:synced/main","+git-annex:synced/git-annex"]
remote: 
remote: Create a new pull request for 'synced/main':        
remote:   https://hub.psychoinformatics.de/www/www-from-model/compare/main...synced/main        
remote: 
remote: 
remote: Create a new pull request for 'synced/git-annex':        
remote:   https://hub.psychoinformatics.de/www/www-from-model/compare/main...synced/git-annex        
remote: 
To https://hub.psychoinformatics.de/www/www-from-model
 * [new branch]      main -> synced/main
 * [new branch]      git-annex -> synced/git-annex
[2026-05-16 09:19:31.549596008] (Utility.Process) process [637] done ExitSuccess
[2026-05-16 09:19:31.55045735] (Utility.Process) process [644] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","push","origin","git-annex"]
[2026-05-16 09:19:31.662607017] (Utility.Process) process [644] done ExitSuccess

ok
[2026-05-16 09:19:31.663427958] (Utility.Process) process [611] done ExitSuccess
[2026-05-16 09:19:31.663999003] (Utility.Process) process [616] done ExitSuccess
push: 1 failed
⚙️ [runner]: exitcode '1': failure

Implemented both using the Fedora location (which also works on Rocky), and falling back to a bundled copy of the CA cert store.

Fedora has no /etc/ssl/certs/ca-certificates.crt. Unlike Rocky, it also has no README saying it should have that file.

On Fedora, both git and git-annex from the standalone bundle fail to find any certs.

(While crypton-x509-system does use the Fedora location, Debian has too old a version for that, and the standalone build inherits that old version.)

So, this is affecting multiple distributions. I would like to find a solution that avoids needing to worry about this class of problems with the standalone build not being sufficiently standalone.

It seems that the best fix would be for the standalone build to use the system CA cert store if one is installed, but fall back to one bundled with it if not. This limits the security exposure of stale certs in the bundle to linux systems where https would otherwise not work without the bundled certs.

I guess this means that the standalone bundle will need to maintain its own list of system cert store locations. Ugh.

Pity that linux does not have some kind of filesystem hierarchy standard. ;-/

But now I've read this in the Changelog, and I could swear it wasn't there 2 days ago

Remove /etc/pki/tls/certs/ca-certificates.crt symlink which was included by mistake

That is not /etc/ssl/certs/ca-certificates.crt so not relevant.

Looking at /etc/ssl/README on Rocky 10, it documents that /etc/ssl/certs is supposed to a) match Debian and b) include ca-certificates.crt.

So Rocky unquestionably has a bug, either a bug in its documentation, or an omission of a file.