I wanted to investigate further and added a credential "helper" that documents what was queried

cat << EOT > /usr/local/bin/git-credential-echo
#!/usr/bin/env bash
exec cat >&2
EOT
chmod +x /usr/local/bin/git-credential-echo
git config --global --add credential.helper echo

I also switched from annex push to annex copy (because this is the aspect that failed). I now see (what I could have seen in the log above already). The issue is not that the credential isn't retrieved properly. It is actually rejected, and the superficial/original error is the result of prompting for another valid credential. Here is the log of a copy call:

git annex --debug copy -t origin .
[2026-05-16 10:01:43.073507233] (Utility.Process) process [639] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","show-ref","git-annex"]
[2026-05-16 10:01:43.075431464] (Utility.Process) process [639] done ExitSuccess
[2026-05-16 10:01:43.075932187] (Utility.Process) process [640] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","show-ref","--hash","refs/heads/git-annex"]
[2026-05-16 10:01:43.077917969] (Utility.Process) process [640] done ExitSuccess
[2026-05-16 10:01:43.078259638] (Annex.Branch) read remote.log
[2026-05-16 10:01:43.079214293] (Utility.Process) process [641] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","cat-file","--batch"]
[2026-05-16 10:01:43.081261767] (Annex.Branch) read proxy.log
[2026-05-16 10:01:43.082419578] (Utility.Process) process [642] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","ls-files","--stage","-z","--error-unmatch","--","."]
[2026-05-16 10:01:43.082798858] (Utility.Process) process [643] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","cat-file","--batch-check=%(objectname) %(objecttype) %(objectsize)","--buffer"]
[2026-05-16 10:01:43.083296281] (Utility.Process) process [644] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","cat-file","--batch=%(objectname) %(objecttype) %(objectsize)","--buffer"]
[2026-05-16 10:01:43.083778403] (Utility.Process) process [641] done ExitSuccess
[2026-05-16 10:01:43.086359241] (Utility.Process) process [645] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","cat-file","--batch=%(objectname) %(objecttype) %(objectsize)","--buffer"]
copy static/graph.json (to origin...) [2026-05-16 10:01:43.210382475] (Utility.Process) process [647] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","-c","filter.annex.smudge=","-c","filter.annex.clean=","-c","filter.annex.process=","write-tree"]
[2026-05-16 10:01:43.214071033] (Utility.Process) process [647] done ExitSuccess
[2026-05-16 10:01:43.214666158] (Utility.Process) process [648] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","show-ref","--hash","refs/annex/last-index"]
[2026-05-16 10:01:43.218272853] (Utility.Process) process [648] done ExitSuccess
[2026-05-16 10:01:43.218310804] (Database.Keys) reconcileStaged start
[2026-05-16 10:01:43.218806637] (Utility.Process) process [649] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","cat-file","--batch-check=%(objectname) %(objecttype) %(objectsize)","--buffer"]
[2026-05-16 10:01:43.219327951] (Utility.Process) process [650] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","cat-file","--batch=%(objectname) %(objecttype) %(objectsize)","--buffer"]
[2026-05-16 10:01:43.219921627] (Utility.Process) process [651] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","-c","filter.annex.smudge=","-c","filter.annex.clean=","-c","filter.annex.process=","-c","diff.external=","diff","ee47aafecb91c163b0eb9e7ef1a35b07d5b1e0b9","8ea4ce9a4065396e07306bc2f30bcf295837ad6f","--raw","-z","--no-abbrev","-G/annex/objects/","--no-renames","--ignore-submodules=all","--no-textconv","--no-ext-diff"]
[2026-05-16 10:01:43.223295855] (Utility.Process) process [651] done ExitSuccess
[2026-05-16 10:01:43.225251807] (Database.Handle) commitDb start
[2026-05-16 10:01:43.225610276] (Database.Handle) commitDb done
[2026-05-16 10:01:43.225676608] (Utility.Process) process [650] done ExitSuccess
[2026-05-16 10:01:43.2257297] (Utility.Process) process [649] done ExitSuccess
[2026-05-16 10:01:43.226178161] (Utility.Process) process [652] call: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","update-ref","refs/annex/last-index","8ea4ce9a4065396e07306bc2f30bcf295837ad6f"]
[2026-05-16 10:01:43.228765129] (Utility.Process) process [652] done ExitSuccess
[2026-05-16 10:01:43.22880699] (Database.Keys) reconcileStaged end
[2026-05-16 10:01:43.246406143] (Utility.Process) process [653] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","credential","fill"]
[2026-05-16 10:01:43.253477499] (Utility.Process) process [653] done ExitSuccess
[2026-05-16 10:01:43.274667127] (Utility.Process) process [656] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","credential","reject"]
protocol=https
host=hub.psychoinformatics.de
username=myuser
password=***
[2026-05-16 10:01:43.2873155] (Utility.Process) process [656] done ExitSuccess
25%   31.98 KiB        70 MiB/s 0s
100%  126.9 KiB       179 MiB/s 0s[2026-05-16 10:01:43.330200019] (Utility.Process) process [662] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","credential","fill"]
protocol=https
host=hub.psychoinformatics.de
fatal: could not read Username for 'https://hub.psychoinformatics.de': No such device or address
[2026-05-16 10:01:43.341958838] (Utility.Process) process [662] done ExitFailure 128

  user error (git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","credential","fill"] exited 128)
[2026-05-16 10:01:43.357710043] (Utility.Process) process [668] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","credential","fill"]
protocol=https
host=hub.psychoinformatics.de
fatal: could not read Username for 'https://hub.psychoinformatics.de': No such device or address
[2026-05-16 10:01:43.369272597] (Utility.Process) process [668] done ExitFailure 128

  user error (git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","credential","fill"] exited 128)

failed
[2026-05-16 10:01:43.371334491] (Utility.Process) process [645] done ExitSuccess
[2026-05-16 10:01:43.371445144] (Utility.Process) process [644] done ExitSuccess
[2026-05-16 10:01:43.371533016] (Utility.Process) process [643] done ExitSuccess
[2026-05-16 10:01:43.371599238] (Utility.Process) process [642] done ExitSuccess
copy: 1 failed

I am now seeing the "same" problem in a different context, where no 3rd-party credential helper is involved.

Concretely, I have a Forgejo actions workflow that manually pre-fills a credential via

git config --global credential.helper cache
git credential approve <<EOF
protocol=https
host=hub.psychoinformatics.de
username=${{ forgejo.actor }}
password=${{ forgejo.token }}
EOF

This appears to be working, as confirmed by

git credential fill
protocol=https
host=hub.psychoinformatics.de

which returns

username=myuser
password=***

However, running git annex push immediately afterwards has the git-push parts working, but the internal credential fill fail with fatal: could not read Username for 'https://hub.psychoinformatics.de': No such device or address (full log below).

+ git annex --debug push
[2026-05-16 09:19:30.185331705] (Utility.Process) process [597] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","show-ref","git-annex"]
[2026-05-16 09:19:30.189155934] (Utility.Process) process [597] done ExitSuccess
[2026-05-16 09:19:30.189885413] (Utility.Process) process [598] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","show-ref","--hash","refs/heads/git-annex"]
[2026-05-16 09:19:30.192024278] (Utility.Process) process [598] done ExitSuccess
[2026-05-16 09:19:30.192633434] (Annex.Branch) read config.log
[2026-05-16 09:19:30.193750572] (Utility.Process) process [599] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","cat-file","--batch"]
[2026-05-16 09:19:30.195842697] (Utility.Process) process [600] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","symbolic-ref","-q","HEAD"]
[2026-05-16 09:19:30.197578872] (Utility.Process) process [600] done ExitSuccess
[2026-05-16 09:19:30.198023783] (Utility.Process) process [601] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","show-ref","refs/heads/main"]
[2026-05-16 09:19:30.19984727] (Utility.Process) process [601] done ExitSuccess
[2026-05-16 09:19:30.200553359] (Utility.Process) process [602] call: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","show-ref","--verify","-q","refs/heads/synced/main"]
[2026-05-16 09:19:30.20216315] (Utility.Process) process [602] done ExitFailure 1
[2026-05-16 09:19:30.202230582] (Annex.Branch) read remote.log
[2026-05-16 09:19:30.202912609] (Annex.Branch) read proxy.log
[2026-05-16 09:19:30.20410367] (Utility.Process) process [603] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","show-ref","git-annex"]
[2026-05-16 09:19:30.20603299] (Utility.Process) process [603] done ExitSuccess
[2026-05-16 09:19:30.206394769] (Utility.Process) process [604] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","show-ref","--hash","refs/heads/git-annex"]
[2026-05-16 09:19:30.208407081] (Utility.Process) process [604] done ExitSuccess
[2026-05-16 09:19:30.209282774] (Utility.Process) process [605] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","show-ref","--hash","refs/heads/git-annex"]
[2026-05-16 09:19:30.211913422] (Utility.Process) process [605] done ExitSuccess
[2026-05-16 09:19:30.212857467] (Utility.Process) process [606] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","log","-z","--pretty=format:%H %ct","--raw","--no-abbrev","--no-renames","--reverse","--follow","668c5aa16d4e9ee0af8fc63f4fbb3e7cfb4b93d4","--","migrate.tree"]
[2026-05-16 09:19:30.215528925] (Utility.Process) process [606] done ExitSuccess
[2026-05-16 09:19:30.216522001] (Utility.Process) process [607] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","ls-files","--stage","-z","--error-unmatch","--"]
[2026-05-16 09:19:30.217095856] (Utility.Process) process [608] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","cat-file","--batch-check=%(objectname) %(objecttype) %(objectsize)","--buffer"]
[2026-05-16 09:19:30.217808254] (Utility.Process) process [609] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","cat-file","--batch=%(objectname) %(objecttype) %(objectsize)","--buffer"]
[2026-05-16 09:19:30.218346408] (Utility.Process) process [599] done ExitSuccess
[2026-05-16 09:19:30.222153637] (Utility.Process) process [610] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","cat-file","--batch=%(objectname) %(objecttype) %(objectsize)","--buffer"]
[2026-05-16 09:19:30.228726336] (Annex.Branch) read trust.log
[2026-05-16 09:19:30.229328452] (Utility.Process) process [611] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","cat-file","--batch"]
[2026-05-16 09:19:30.230981234] (Annex.Branch) read cluster.log
[2026-05-16 09:19:30.237149214] (Utility.Process) process [614] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","-c","filter.annex.smudge=","-c","filter.annex.clean=","-c","filter.annex.process=","write-tree"]
[2026-05-16 09:19:30.239369771] (Utility.Process) process [614] done ExitSuccess
[2026-05-16 09:19:30.239845993] (Utility.Process) process [615] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","show-ref","--hash","refs/annex/last-index"]
[2026-05-16 09:19:30.241848385] (Utility.Process) process [615] done ExitSuccess
[2026-05-16 09:19:30.242685967] (Utility.Process) process [616] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","check-attr","-z","--stdin","annex.backend","annex.largefiles","annex.numcopies","annex.mincopies","--"]
[2026-05-16 09:19:30.244064203] (Annex.Branch) read numcopies.log
[2026-05-16 09:19:30.244322359] (Annex.Branch) read mincopies.log
[2026-05-16 09:19:30.257332835] (Annex.Branch) read group.log
[2026-05-16 09:19:30.25750822] (Annex.Branch) read group-preferred-content.log
[2026-05-16 09:19:30.257716505] (Annex.Branch) read preferred-content.log
[2026-05-16 09:19:30.257939851] (Annex.Branch) read required-content.log
[2026-05-16 09:19:30.258732681] (Annex.Branch) read 455/e88/MD5E-s129943--06caaa82dc6cfdd358085974adbbe8d3.json.log
copy static/graph.json (to origin...) [2026-05-16 09:19:30.378214958] (Utility.Process) process [617] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","credential","fill"]
[2026-05-16 09:19:30.383551986] (Utility.Process) process [617] done ExitSuccess
[2026-05-16 09:19:30.404495517] (Utility.Process) process [620] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","credential","reject"]
[2026-05-16 09:19:30.40926853] (Utility.Process) process [620] done ExitSuccess
25%   31.98 KiB        68 MiB/s 0s
100%  126.9 KiB       177 MiB/s 0s[2026-05-16 09:19:30.450818263] (Utility.Process) process [623] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","credential","fill"]
fatal: could not read Username for 'https://hub.psychoinformatics.de': No such device or address
[2026-05-16 09:19:30.455457733] (Utility.Process) process [623] done ExitFailure 128

  user error (git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","credential","fill"] exited 128)
[2026-05-16 09:19:30.473680804] (Utility.Process) process [626] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","credential","fill"]
fatal: could not read Username for 'https://hub.psychoinformatics.de': No such device or address
[2026-05-16 09:19:30.47933976] (Utility.Process) process [626] done ExitFailure 128

  user error (git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","credential","fill"] exited 128)

failed
[2026-05-16 09:19:30.481071965] (Utility.Process) process [610] done ExitSuccess
[2026-05-16 09:19:30.481128386] (Utility.Process) process [609] done ExitSuccess
[2026-05-16 09:19:30.481169867] (Utility.Process) process [608] done ExitSuccess
[2026-05-16 09:19:30.481201948] (Utility.Process) process [607] done ExitSuccess
[2026-05-16 09:19:30.481629439] (Utility.Process) process [629] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","symbolic-ref","-q","HEAD"]
[2026-05-16 09:19:30.483084607] (Utility.Process) process [629] done ExitSuccess
[2026-05-16 09:19:30.483620051] (Utility.Process) process [630] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","show-ref","refs/heads/main"]
[2026-05-16 09:19:30.486919556] (Utility.Process) process [630] done ExitSuccess
[2026-05-16 09:19:30.487433589] (Utility.Process) process [631] call: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","branch","-f","synced/main","refs/heads/main"]
[2026-05-16 09:19:30.490124219] (Utility.Process) process [631] done ExitSuccess
[2026-05-16 09:19:30.490606621] (Utility.Process) process [632] call: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","show-ref","--verify","-q","refs/remotes/origin/synced/main"]
[2026-05-16 09:19:30.493191278] (Utility.Process) process [632] done ExitFailure 1

push origin 

[2026-05-16 09:19:30.493862885] (Utility.Process) process [633] call: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","push","origin","main"]
Everything up-to-date
[2026-05-16 09:19:30.622077877] (Utility.Process) process [633] done ExitSuccess
[2026-05-16 09:19:30.622889428] (Utility.Process) process [637] call: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","push","origin","main:synced/main","+git-annex:synced/git-annex"]
remote: 
remote: Create a new pull request for 'synced/main':        
remote:   https://hub.psychoinformatics.de/www/www-from-model/compare/main...synced/main        
remote: 
remote: 
remote: Create a new pull request for 'synced/git-annex':        
remote:   https://hub.psychoinformatics.de/www/www-from-model/compare/main...synced/git-annex        
remote: 
To https://hub.psychoinformatics.de/www/www-from-model
 * [new branch]      main -> synced/main
 * [new branch]      git-annex -> synced/git-annex
[2026-05-16 09:19:31.549596008] (Utility.Process) process [637] done ExitSuccess
[2026-05-16 09:19:31.55045735] (Utility.Process) process [644] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","push","origin","git-annex"]
[2026-05-16 09:19:31.662607017] (Utility.Process) process [644] done ExitSuccess

ok
[2026-05-16 09:19:31.663427958] (Utility.Process) process [611] done ExitSuccess
[2026-05-16 09:19:31.663999003] (Utility.Process) process [616] done ExitSuccess
push: 1 failed
⚙️ [runner]: exitcode '1': failure

Implemented both using the Fedora location (which also works on Rocky), and falling back to a bundled copy of the CA cert store.

Fedora has no /etc/ssl/certs/ca-certificates.crt. Unlike Rocky, it also has no README saying it should have that file.

On Fedora, both git and git-annex from the standalone bundle fail to find any certs.

(While crypton-x509-system does use the Fedora location, Debian has too old a version for that, and the standalone build inherits that old version.)

So, this is affecting multiple distributions. I would like to find a solution that avoids needing to worry about this class of problems with the standalone build not being sufficiently standalone.

It seems that the best fix would be for the standalone build to use the system CA cert store if one is installed, but fall back to one bundled with it if not. This limits the security exposure of stale certs in the bundle to linux systems where https would otherwise not work without the bundled certs.

I guess this means that the standalone bundle will need to maintain its own list of system cert store locations. Ugh.

Pity that linux does not have some kind of filesystem hierarchy standard. ;-/

But now I've read this in the Changelog, and I could swear it wasn't there 2 days ago

Remove /etc/pki/tls/certs/ca-certificates.crt symlink which was included by mistake

That is not /etc/ssl/certs/ca-certificates.crt so not relevant.

Looking at /etc/ssl/README on Rocky 10, it documents that /etc/ssl/certs is supposed to a) match Debian and b) include ca-certificates.crt.

So Rocky unquestionably has a bug, either a bug in its documentation, or an omission of a file.

It seems like a good idea for git-annex to support the same configuration as git does for the CA cert locations. So I've made git-annex support GIT_SSL_CAINFO (etc) for its own https connections.

That doesn't address the bug, but it does mean that generally, if the linux standalone build is having problems with finding system CA certs, setting one environment variable is enough to tell both git and git-annex where to look.

Are you sure the image you checked has a new enough version of the package?

Yes. I've used a freshly pulled docker.io/rockylinux/rockylinux:9 and docker.io/rockylinux/rockylinux:10 respectively and then ran dnf update too. dnf lists the installed version as 2025.2.80_v9.0.305-102.el10. I've even checked AlmaLinux (docker.io/library/almalinux:10) too now, and it is the same thing there.

But now I've read this in the Changelog, and I could swear it wasn't there 2 days ago: https://rockylinux.pkgs.org/10/rockylinux-baseos-x86_64/ca-certificates-2025.2.80_v9.0.305-102.el10.noarch.rpm.html#:~:text=%2D%20Remove%20%2Fetc%2Fpki%2Ftls%2Fcerts%2Fca%2Dcertificates%2Ecrt%20symlink%20which%20was%20included%20by%20mistake. So it seems the file is not intended to be there.

That happens to read every cert file in /etc/ssl/certs/ (amoung some other places for other OS's, including /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem for fedora), which would include /etc/ssl/certs/ca-bundle.crt.

That's good to know! Then I think it should be enough to set GIT_SSL_CAINFO to get things working. Maybe git-annex-standalone's runshell could try to discover the certificate location and set GIT_SSL_CAINFO if necessary? I.e. if GIT_SSL_CAINFO is unset check that /etc/ssl/certs/ca-certificates.crt exists, if it doesn't check for ca-bundle.crt, if that does exist set GIT_SSL_CAINFO=/etc/ssl/certs/ca-bundle.crt, otherwise log a warning.

git-annex's own SSL cert checking comes via http-client and crypton-connection, which uses crypton-x509-system.

That happens to read every cert file in /etc/ssl/certs/ (amoung some other places for other OS's, including /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem for fedora), which would include /etc/ssl/certs/ca-bundle.crt.

2025-08-18 - Frantisek Krenzelok <fkrenzel@redhat.com> - 2024.2.69_v8.0.303-102.5
- Bring back openssl trusted format bundle as well (Resolves: RHEL-109484)
- Restored the following symlinks:
* /etc/pki/tls/cert.pem
* /etc/pki/tls/certs/ca-certificates.crt

That certainly reads as if the package is intended to contain the file, and is buggy.

Are you sure the image you checked has a new enough version of the package?