Please describe the problem.
links: prior report/fix of testing on beegfs 4 years ago; different site/version
Currently I observed 35 tests failing (now at a potential repronim reprocenter location)
yarick@ducky:/data/mri_dicom/tmp/test-git-annex
*$> grep FAIL .duct/logs/2025.08.28T21.04.10-56898_stdout | nl | tail
26 add: FAIL (2.30s)
27 add: FAIL (2.34s)
28 add: FAIL (2.80s)
29 add: FAIL (2.21s)
30 add: FAIL (1.98s)
31 add: FAIL (3.16s)
32 add: FAIL (4.27s)
33 git-remote-annex exporttree: FAIL (8.45s)
34 export and import: FAIL (10.40s)
35 export and import of subdir: FAIL (15.99s)
full log: http://www.oneukrainian.com/tmp/2025.08.28T21.04.10-56898_stdout
some info:
$> modinfo beegfs
filename: /lib/modules/5.15.0-122-generic/updates/fs/beegfs_autobuild/beegfs.ko
version: 7.4.6
alias: fs-beegfs
author: ThinkParQ GmbH
description: BeeGFS parallel file system client (https://www.beegfs.io)
license: GPL v2
srcversion: 9F666198EABF0EB756ED3AC
depends: ib_core,rdma_cm
retpoline: Y
name: beegfs
vermagic: 5.15.0-122-generic SMP mod_unload modversions
$> mount | grep data/mri_dicom
beegfs_nodev on /data/mri_dicom type beegfs (rw,nodev,relatime,cfgFile=/etc/beegfs/beegfs-client.conf)
What steps will reproduce the problem?
Run tests on beegfs?
What version of git-annex are you using? On what operating system?
*$> git annex version
git-annex version: 10.20250721-g8867e7590a3a70afa8a93d2fefab94adc9a176d0
build flags: Assistant Webapp Pairing Inotify TorrentParser MagicMime Servant Benchmark Feeds Testsuite S3 WebDAV
dependency versions: aws-0.24.4 bloomfilter-2.0.1.2 crypton-1.0.4 DAV-1.3.4 feed-1.3.2.1 ghc-9.10.1 http-client-0.7.19 persistent
-sqlite-2.13.3.0 torrent-10000.1.3 uuid-1.3.16 yesod-1.6.2.1
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E S
HA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2
B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S
224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL GITBUNDLE GITMANIFEST VURL X*
remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs httpalso borg rclone hoo
k external compute mask
operating system: linux x86_64
supported repository versions: 8 9 10
upgrade supported from repository versions: 0 1 2 3 4 5 6 7 8 9 10
Please provide any additional information below.
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log
# End of transcript or log.
The failures are mostly of two varieties.
type A:
type B:
In both cases, a
git-annex addis succeeding, but the annex objects directory is somehow not getting populated. Or at least, a subsequent read of a file in it has the filesystem not knowing the file that the add put there is there.It seems quite likely a lot of other tests would also fail, but they are being skipped because the add tests fail.
In one case, the add tests are succeeding (on an adjusted unlocked branch), but then a subsequent test fails:
I'm not familiar with beegfs, but its documentation such as this https://doc.beegfs.io/latest/architecture/overview.html makes me wonder if it manages to behave consistently as we would expect a filesystem to behave.
In particular, we have a file being moved from one directory to another directory. Beegfs's docs says it will pick a random metadata node for each directory. So there can be two metadata nodes that have to be updated for a move. If one node somehow lags seeing the update, could that result in the file not appearing as present in the destination directory after the move?
I'm only speculating about how beegfs might work, but it seems unlikely that git-annex has a bug that causes it to lose an annexed file when all it's done is move it to the objects directory, that only manifests on this one filesystem.
A good next step might be to try manually adding an annexed file and see if there is some lag between
git-annex addand being able to read the content of the symlink. Eg, compare:eh, it is indeed quite a f...un filesystem: even
chmodmight endup unhappyand it might indeed take minute(s) for filesytem to become "consistent" with expected state. And it seems that even a minute might not be enough!!!
with this full tune up script
it
Waited 450 seconds for bar to appear... and on this funny system bash would report that file is available so symlink would not be broken to those tests:I will report to sysadmins -- may be they have ideas/feedback
admins tamed the beast down (disabled some "optimization" options which were enabled) and the beast started to behave better -- now just
fails. Full log link. That is still with the same 10.20250721-g8867e7590a3a70afa8a93d2fefab94adc9a176d0
All the fails now look like this:
This is the same kind of EBUSY problem as on the previous Beegfs bug report. In that report I hypothesized that Beegfs might not like an open file to be renamed. I'm not sure if we ever verified my fixes in that one fixed a problem with Beegfs, but it still seems like a good hypothesis.
"export.ex/" is a log file that git-annex uses to keep track of files that were part of a tree exported to a special remote, but that were excluded from the export by its preferred content settings. To populate that file, git-annex opens a temp file, writes to it as the export runs, then closes it and renames it.
If the rename() fails, it falls back to trying "mv", which is why there are also "mv" errors in the transcript above. Anyway, I've verified the FD is closed before that point.
But, the "..." includes some fork and exec. And this FD is never set close-on-exec! And the processes started while it's open include "git cat-file --batch", which is a long-running process that will still be left running when the rename happens.
This was pretty surprising to me, I did not realize git-annex was generally leaking FDs to child processes in this way. It's easy to demonstrate with a simpler program:
So, really supporting this would mean auditing every file git-annex opens with openFile to see if the handle is ever passed to a child process, and otherwise making it use CloseOnExec.
I don't care a great deal about supporting Beegfs; it would be nice to support it in some of its less crazy configurations if possible. But not leaking FDs while running child processes seems like something that ought to be fixed for other reasons.
openFileis not the only one that would need to be dealt with. AlsowithFile,openBinaryFile,withBinaryFile,appendFile, andopenTempFile,readFile, andwriteFile(includingL.versions).Since none of those provide a way to set CloseOnExec, they would have to be changed to be implemented using
openFdwith CloseOnExec, and then mkHandleFromFD. Or rewritten to use https://hackage.haskell.org/package/file-io-0.1.5/docs/src/System.File.OsPath.Internal.html#openFileWithCloseOnExecI have checked and none of those are ever used to create a handle that is intentionally passed to a child process. The only uses of
handleToFdresult in a FD that gets dupped to another FD number, and dup() does not inherit the close-on-exec flag. So it should be safe to just write new versions of all of those.Also there are a few uses of
openFdthat don't set CloseOnExec.I've checked all uses of openFd and made CloseOnExec be used where appropriate. Also checked all the fd dupping.
This won't fix the problem, but is necessary goundwork for fixing openFile and the rest.
There is also the problem that any haskell library that does anything with a file might use any of the above internally without setting close-on-exec.
For example, opening a https connection can result in readFile opening a handle to a file in /etc/ssl/certs/, which will not be closed on exec. And which can leak out via another thread doing an exec at just the right time.
But inheriting a single FD like that is not going to cause problems for beegfs or anything else.
The ones I'd worry about is if a haskell library is doing something with a file in the git-annex repo.
Most dependencies of git-annex clearly don't open files there, and most open no files at all. Ones I need to check:
robust_opensets the close-on-exec flag)At this point, I'm reasonably satisfied about libraries not causing the problem.
copyFile also leaks a FD. git-annex only uses it on Windows though, so not a problem.
As far as general correctness goes, I've reported a bug upstream about it https://github.com/haskell/directory/issues/203, which I suspect they'll fix eventually since they have fixed similar problems before.
Also filed https://github.com/haskell/file-io/issues/44 which would avoid git-annex needing to reimplement a lot of this stuff.
I've made all OsPath operations in git-annex set the close-on-exec flag.
This will largely fix the problem when git-annex is built with the OsPath build flag. Which has now become default, but it's still possible for git-annex to be built without that build flag. Check
git-annex versionfor "OsPath" to know if your git-annex was built with this. I think it will probably pass the test suite on beegfs at this point.(However, the openTempFile implementation sets the flag only after opening the file, so is still vulnerable to a race where the fd can leak to an execed process. That could use more work, but at least is a quite narrow race window.)
Also the few ByteString readFile/writeFile calls were converted to the OsPath operations.
What remains to be done to fully fix this:
So, this bug is now tied to OsPath conversion in 2 ways.
I've now converted all functions listed above to ones that set the close-on-exec flag. (When building with the OsPath flag.)
And checked all the libraries in comment #8.
All that remains is the openTempFile race.
There are a few other races where a FD can leak to a child process still, where
setFdOptionis used after dup()/pipe().These don't involve files, so won't affect beegfs. They should still be fixed.
While dup3() and pipe2() allow setting the close-on-exec flag, I don't think they're portable enough to rely on.
It may be possible to rewrite these handful of things to avoid the problem:
Alternatively, it would be possible to solve all of these issues, as well as the openTempFile race, by making the wrappers in Utility.Process prevent starting a process when a global MVar is empty. And have a function that runs an action with the MVar emptied. Then the call to dup/pipe could run effectively atomically with the
setFdOption. There would be a small overhead in checking the MVar on each exec, but probably too small to be noticable.There is some potential for any of these FD leaks to compromise security, in cases where a child process ends up running untrusted code in some kind of sandbox, that is sufficiently leaky that the leaked FD is accessible inside the sandbox.
I don't think any existing special remote or other git-annex addons behave that way, so don't think this is an exploitable security hole. Arguably, if sandboxing untrusted code, it's on you to avoid exposing open Fds to it.
However, since security is involved, it does need to be fixed comprehensively in git-annex, including the remaining races.
I have reran with freshish build 10.20250828+git58-g38786a4e5e-1~ndall+1 and still observe those FAILs as before IIRC
Drat. Reopened bug.
Is the error for these still "export.ex [...] Device or resource busy"?
If so, the problem must not be beegfs not liking an open file to be renamed, but something else.
I have verified that the temp file that gets renamed to the "export.ex" log file is now opened with
O_CLOEXEC.Confirmed in your log that git-annex is not built with the OsPath build flag, so it will still not be using
O_CLOEXEC.It would be good to get a build with OsPath and test it to see if my fixes actually did work. Debian doesn't include the necessary library yet, so a build using stack is needed.
I have enabled OsPath in the build at https://downloads.kitenet.net/git-annex/autobuild/amd64/git-annex-standalone-amd64.tar.gz
that version had no errors:
All tests succeeded. (Ran 50 test groups in 11m54s)So, this will be default option? any specific dependency requirements we need to add/constrain?
Yay!
OsPath needs the os-string and file-io haskell packages. Which are not currently in Debian. So either work will need to be done to package those, or when Debian upgrades ghc to 9.12.2, it will include those libraries automatically since they are bundled with ghc since that version.
Maybe you know more than I do about the state of Debian's haskell support.
The transition is being tracked at RawFilePath conversion but I don't know yet what the solution is to getting the dependencies broadly available.
(Or I could implement the same fixes when not built with that flag of course. It is doable. Just annoying especially since that code will have to be carefully gotten just right, only to be thrown away later.)
The pypi whl builds should already have it, as all stack builds default to having the OsPath build flag enabled already.