Please describe the problem.
On Rocky Linux 9 (and therefore on the HPC Systems at FZJ) with git-annex installed using the standalone package (either directly or through pixi/conda-forge) I am getting this error message from git annex pull:
# git annex pull origin --no-content
pull origin
fatal: unable to access 'https://hub.datalad.org/distribits/recordings.git/': Problem with the SSL CA cert (path? access rights?)
ok
What steps will reproduce the problem?
dnf update -y
dnf install -y wget git
cd /opt
wget https://downloads.kitenet.net/git-annex/linux/current/git-annex-standalone-amd64.tar.gz
tar xzvf git-annex-standalone-amd64.tar.gz
ln -s /opt/git-annex.linux/git-annex /usr/local/bin
cd /tmp
git config --global user.email "you@example.com"
git config --global user.name "Your Name"
git clone https://hub.datalad.org/distribits/recordings.git
cd recordings
git annex pull origin --no-content
Saved as reproducer.bash this can be easily tested using:
podman run --rm -it docker.io/library/rockylinux:9 bash < reproducer.bash
What version of git-annex are you using? On what operating system?
# git annex version
git-annex version: 10.20260420-g74cffe2f348afd81711fa9f69e165e04e11f1ef0
build flags: Assistant Webapp Inotify DBus DesktopNotify TorrentParser MagicMime Benchmark Feeds Testsuite S3 WebDAV Servant OsPath
dependency versions: aws-0.24.4 bloomfilter-2.0.1.3 crypton-1.0.4 DAV-1.3.4 feed-1.3.2.1 ghc-9.10.3 http-client-0.7.19 torrent-10000.1.3 uuid-1.3.16 yesod-1.6.2.1
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL GITBUNDLE GITMANIFEST VURL X*
remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs httpalso borg rclone hook external compute mask
operating system: linux x86_64
supported repository versions: 8 9 10
upgrade supported from repository versions: 0 1 2 3 4 5 6 7 8 9 10
local repository version: 10
on Rocky Linux 9, reproduced in a podman container as mentioned above.
Please provide any additional information below.
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log
# End of transcript or log.
The ca-certificates.crt file seems to be hardcoded in the git-annex-standalone package:
The standalone build is made on Debian and inherits where it expects to find SSL certs from there. On Debian, the ca-certificates package provides the certificates, and derives from those shipped with Mozilla.
Rocky does have an equivilant ca-certificates package, which seems likely to provide certificates in a place that will work.
There may be an argument for the standalone build bundling its own copy of certificates. Of course, it then would need a security update every time there is a removal. But, Debian allows the admin to configure which certificate authorities they trust and only populates the file with those. If the standalone build overrode that, it would be at least surprising. It might be a good compromise for it to ship its own copy, but only use it if the file is not present in /etc/.
As well as the 2 C libraries, git-annex links to a haskell library that reads the certificates too. Each of these would need to be patched, probably, I don't think there is any way to override the location with an environment variable.
Unfortunately that is not the case. The ca-certificates package on Rocky Linux 9 does not provide /etc/ssl/certs/ca-certificates.crt, only /etc/ssl/certs/ca-bundle.crt and /etc/ssl/ca-bundle.trust.crt. /etc/ssl/certs/ca-bundle.crt seems to be Rocky's equivalent to /etc/ssl/certs/ca-certificates.crt, but it is not picked up by the git-annex-standalone package.
I can confirm that symlinking (
ln -s /etc/ssl/certs/ca-bundle.crt /etc/ssl/certs/ca-certificates.crt) fixes the issue, but I am not sure if I will be able to convince the HPC sysadmins to do so everywhere.While the changelog for the Rocky Linux 10 ca-certificates package says that the /etc/ssl/certs/ca-certificates.crt symlink has been restored (https://rockylinux.pkgs.org/10/rockylinux-baseos-x86_64/ca-certificates-2025.2.80_v9.0.305-102.el10.noarch.rpm.html#:~:text=%2A%20%2Fetc%2Fssl%2Fcerts%2Fca%2Dcertificates%2Ecrt) this doesn't actually seem to be the case, at least I can't find it in the docker.io/rockylinux/rockylinux:10 image.
At least for GnuTLS it seems to be not implemented: https://gitlab.com/gnutls/gnutls/-/work_items/1279. But some applications make it overwritable, and this gave me another idea: setting
export GIT_SSL_CAINFO=/etc/ssl/certs/ca-bundle.crteliminates the error I was seeing (which seemed to originate from git after all). But I am not sure if there won't be other issues due to the missing /etc/ssl/certs/ca-certificates.crt file and that path being hardcoded in GnuTLS. At leastgit annex getvia p2phttp seems to be working fine, but now I am wondering which CAs it is actually using to validate the remote...That certainly reads as if the package is intended to contain the file, and is buggy.
Are you sure the image you checked has a new enough version of the package?
git-annex's own SSL cert checking comes via http-client and crypton-connection, which uses crypton-x509-system.
That happens to read every cert file in
/etc/ssl/certs/(amoung some other places for other OS's, including/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pemfor fedora), which would include/etc/ssl/certs/ca-bundle.crt.Yes. I've used a freshly pulled docker.io/rockylinux/rockylinux:9 and docker.io/rockylinux/rockylinux:10 respectively and then ran
dnf updatetoo. dnf lists the installed version as2025.2.80_v9.0.305-102.el10. I've even checked AlmaLinux (docker.io/library/almalinux:10) too now, and it is the same thing there.But now I've read this in the Changelog, and I could swear it wasn't there 2 days ago: https://rockylinux.pkgs.org/10/rockylinux-baseos-x86_64/ca-certificates-2025.2.80_v9.0.305-102.el10.noarch.rpm.html#:~:text=%2D%20Remove%20%2Fetc%2Fpki%2Ftls%2Fcerts%2Fca%2Dcertificates%2Ecrt%20symlink%20which%20was%20included%20by%20mistake. So it seems the file is not intended to be there.
That's good to know! Then I think it should be enough to set
GIT_SSL_CAINFOto get things working. Maybe git-annex-standalone's runshell could try to discover the certificate location and setGIT_SSL_CAINFOif necessary? I.e. ifGIT_SSL_CAINFOis unset check that/etc/ssl/certs/ca-certificates.crtexists, if it doesn't check for ca-bundle.crt, if that does exist setGIT_SSL_CAINFO=/etc/ssl/certs/ca-bundle.crt, otherwise log a warning.It seems like a good idea for git-annex to support the same configuration as git does for the CA cert locations. So I've made git-annex support
GIT_SSL_CAINFO(etc) for its own https connections.That doesn't address the bug, but it does mean that generally, if the linux standalone build is having problems with finding system CA certs, setting one environment variable is enough to tell both git and git-annex where to look.
That is not /etc/ssl/certs/ca-certificates.crt so not relevant.
Looking at /etc/ssl/README on Rocky 10, it documents that /etc/ssl/certs is supposed to a) match Debian and b) include ca-certificates.crt.
So Rocky unquestionably has a bug, either a bug in its documentation, or an omission of a file.
Fedora has no /etc/ssl/certs/ca-certificates.crt. Unlike Rocky, it also has no README saying it should have that file.
On Fedora, both git and git-annex from the standalone bundle fail to find any certs.
(While crypton-x509-system does use the Fedora location, Debian has too old a version for that, and the standalone build inherits that old version.)
So, this is affecting multiple distributions. I would like to find a solution that avoids needing to worry about this class of problems with the standalone build not being sufficiently standalone.
It seems that the best fix would be for the standalone build to use the system CA cert store if one is installed, but fall back to one bundled with it if not. This limits the security exposure of stale certs in the bundle to linux systems where https would otherwise not work without the bundled certs.
I guess this means that the standalone bundle will need to maintain its own list of system cert store locations. Ugh.
Pity that linux does not have some kind of filesystem hierarchy standard. ;-/
Implemented both using the Fedora location (which also works on Rocky), and falling back to a bundled copy of the CA cert store.