Are you sure the image you checked has a new enough version of the package?

Yes. I've used a freshly pulled docker.io/rockylinux/rockylinux:9 and docker.io/rockylinux/rockylinux:10 respectively and then ran dnf update too. dnf lists the installed version as 2025.2.80_v9.0.305-102.el10. I've even checked AlmaLinux (docker.io/library/almalinux:10) too now, and it is the same thing there.

But now I've read this in the Changelog, and I could swear it wasn't there 2 days ago: https://rockylinux.pkgs.org/10/rockylinux-baseos-x86_64/ca-certificates-2025.2.80_v9.0.305-102.el10.noarch.rpm.html#:~:text=%2D%20Remove%20%2Fetc%2Fpki%2Ftls%2Fcerts%2Fca%2Dcertificates%2Ecrt%20symlink%20which%20was%20included%20by%20mistake. So it seems the file is not intended to be there.

That happens to read every cert file in /etc/ssl/certs/ (amoung some other places for other OS's, including /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem for fedora), which would include /etc/ssl/certs/ca-bundle.crt.

That's good to know! Then I think it should be enough to set GIT_SSL_CAINFO to get things working. Maybe git-annex-standalone's runshell could try to discover the certificate location and set GIT_SSL_CAINFO if necessary? I.e. if GIT_SSL_CAINFO is unset check that /etc/ssl/certs/ca-certificates.crt exists, if it doesn't check for ca-bundle.crt, if that does exist set GIT_SSL_CAINFO=/etc/ssl/certs/ca-bundle.crt, otherwise log a warning.

git-annex's own SSL cert checking comes via http-client and crypton-connection, which uses crypton-x509-system.

That happens to read every cert file in /etc/ssl/certs/ (amoung some other places for other OS's, including /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem for fedora), which would include /etc/ssl/certs/ca-bundle.crt.

2025-08-18 - Frantisek Krenzelok <fkrenzel@redhat.com> - 2024.2.69_v8.0.303-102.5
- Bring back openssl trusted format bundle as well (Resolves: RHEL-109484)
- Restored the following symlinks:
* /etc/pki/tls/cert.pem
* /etc/pki/tls/certs/ca-certificates.crt

That certainly reads as if the package is intended to contain the file, and is buggy.

Are you sure the image you checked has a new enough version of the package?

Rocky does have an equivilant ca-certificates package, which seems likely to provide certificates in a place that will work.

Unfortunately that is not the case. The ca-certificates package on Rocky Linux 9 does not provide /etc/ssl/certs/ca-certificates.crt, only /etc/ssl/certs/ca-bundle.crt and /etc/ssl/ca-bundle.trust.crt. /etc/ssl/certs/ca-bundle.crt seems to be Rocky's equivalent to /etc/ssl/certs/ca-certificates.crt, but it is not picked up by the git-annex-standalone package.

I can confirm that symlinking (ln -s /etc/ssl/certs/ca-bundle.crt /etc/ssl/certs/ca-certificates.crt) fixes the issue, but I am not sure if I will be able to convince the HPC sysadmins to do so everywhere.

While the changelog for the Rocky Linux 10 ca-certificates package says that the /etc/ssl/certs/ca-certificates.crt symlink has been restored (https://rockylinux.pkgs.org/10/rockylinux-baseos-x86_64/ca-certificates-2025.2.80_v9.0.305-102.el10.noarch.rpm.html#:~:text=%2A%20%2Fetc%2Fssl%2Fcerts%2Fca%2Dcertificates%2Ecrt) this doesn't actually seem to be the case, at least I can't find it in the docker.io/rockylinux/rockylinux:10 image.

I don't think there is any way to override the location with an environment variable.

At least for GnuTLS it seems to be not implemented: https://gitlab.com/gnutls/gnutls/-/work_items/1279. But some applications make it overwritable, and this gave me another idea: setting export GIT_SSL_CAINFO=/etc/ssl/certs/ca-bundle.crt eliminates the error I was seeing (which seemed to originate from git after all). But I am not sure if there won't be other issues due to the missing /etc/ssl/certs/ca-certificates.crt file and that path being hardcoded in GnuTLS. At least git annex get via p2phttp seems to be working fine, but now I am wondering which CAs it is actually using to validate the remote...

The standalone build is made on Debian and inherits where it expects to find SSL certs from there. On Debian, the ca-certificates package provides the certificates, and derives from those shipped with Mozilla.

Rocky does have an equivilant ca-certificates package, which seems likely to provide certificates in a place that will work.

There may be an argument for the standalone build bundling its own copy of certificates. Of course, it then would need a security update every time there is a removal. But, Debian allows the admin to configure which certificate authorities they trust and only populates the file with those. If the standalone build overrode that, it would be at least surprising. It might be a good compromise for it to ship its own copy, but only use it if the file is not present in /etc/.

As well as the 2 C libraries, git-annex links to a haskell library that reads the certificates too. Each of these would need to be patched, probably, I don't think there is any way to override the location with an environment variable.

datalad push wants to use the same git push operations as git-annex push does, which is nontrivial to reimplement, especially in its handling of the git-annex branch. See the long comment on pushBranch explaining the order of operations.

This is one place where git-annex push can't be emulated using other git-annex commands that do support --json.

But, git-annex push --no-content doesn't do much besides run pushBranch. So datalad push could use it when run in a git-annex repository. There's no need for it to support --json either, the regular git push output goes to stderr, so it can parse the git push progress out of stderr as before.

It may want to pass --quiet to avoid the usual git-annex output to stdout. AFAICS, git push does not itself output to stdout.

The only other thing that command does besides pushBranch is updateBranches, which updates view branches and adjusted branches when run in one.

git pull outputs its progress to stderr. So --json could leave that alone and a program wanting to parse it just consume stderr. Delimiters could be added to stderr around the git pull (with a separate option) to make it easier for a program to find and parse it.

git pull also outputs some things to stdout. In particular, that includes the git merge output when the merge is successful. It seems to me that could be put in the json object, eg:

{"command":"pull","output":["Updating 8a433d0..9d47770" ...

While that will buffer it until the pull is complete. That seems ok; it's displayed by git pull after the usually more expensive network operation, so buffering it briefly wouldn't be too noticable if a json consumer chooses to show it to the user.

Note that git-annex pull will pull from the remote a second time after transferring content to/from it. So the json will have 2 "command":"pull" records. And stderr may contain 2 delimited git pull stderrs. The --json consumer may find that surprising, and it doesn't always happen, which gets back to the original problem of the --json not being discoverable.

In the case where the git operation needs to prompt for a password, this would leave the user with a password prompt with no prior indication of what is being done. I don't think that's acceptable.

git pull and git push over ssh prompt for the password (to /dev/tty) before outputing anything else. So I suppose it is acceptable.

The ca-certificates.crt file seems to be hardcoded in the git-annex-standalone package:

$ grep -R ca-certificates.crt .
grep: ./usr/lib/x86_64-linux-gnu/tls/x86_64: Warnung: rekursive Verzeichnisschleife
grep: ./usr/lib/x86_64-linux-gnu/glibc-hwcaps/x86-64-v3: Warnung: rekursive Verzeichnisschleife
grep: ./usr/lib/x86_64-linux-gnu/glibc-hwcaps/x86-64-v2: Warnung: rekursive Verzeichnisschleife
grep: ./usr/lib/x86_64-linux-gnu/glibc-hwcaps/x86-64-v4: Warnung: rekursive Verzeichnisschleife
grep: ./usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4: Übereinstimmungen in Binärdatei
grep: ./usr/lib/x86_64-linux-gnu/libgnutls.so.30: Übereinstimmungen in Binärdatei
grep: ./usr/lib/x86_64-linux-gnu/x86_64: Warnung: rekursive Verzeichnisschleife
$ strings ./usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4 | grep ca-certificates
/etc/ssl/certs/ca-certificates.crt
$ strings ./usr/lib/x86_64-linux-gnu/libgnutls.so.30 | grep ca-certificates
/etc/ssl/certs/ca-certificates.crt