35 failed tests on beegfs

Please describe the problem.

links: prior report/fix of testing on beegfs 4 years ago; different site/version

Currently I observed 35 tests failing

yarick@ducky:/data/mri_dicom/tmp/test-git-annex
*$> grep FAIL .duct/logs/2025.08.28T21.04.10-56898_stdout  | nl | tail
    26        add:                           FAIL (2.30s)
    27        add:                                 FAIL (2.34s)
    28        add:                              FAIL (2.80s)
    29        add:                                 FAIL (2.21s)
    30        add:                                    FAIL (1.98s)
    31        add:               FAIL (3.16s)
    32        add:                                                FAIL (4.27s)
    33      git-remote-annex exporttree:      FAIL (8.45s)
    34      export and import:                     FAIL (10.40s)
    35      export and import of subdir:           FAIL (15.99s)

full log: http://www.oneukrainian.com/tmp/2025.08.28T21.04.10-56898_stdout

some info:

$> modinfo beegfs
filename:       /lib/modules/5.15.0-122-generic/updates/fs/beegfs_autobuild/beegfs.ko
version:        7.4.6
alias:          fs-beegfs
author:         ThinkParQ GmbH
description:    BeeGFS parallel file system client (https://www.beegfs.io)
license:        GPL v2
srcversion:     9F666198EABF0EB756ED3AC
depends:        ib_core,rdma_cm
retpoline:      Y
name:           beegfs
vermagic:       5.15.0-122-generic SMP mod_unload modversions 

$> mount | grep data/mri_dicom
beegfs_nodev on /data/mri_dicom type beegfs (rw,nodev,relatime,cfgFile=/etc/beegfs/beegfs-client.conf)

What steps will reproduce the problem?

Run tests on beegfs?

What version of git-annex are you using? On what operating system?

*$> git annex version
git-annex version: 10.20250721-g8867e7590a3a70afa8a93d2fefab94adc9a176d0
build flags: Assistant Webapp Pairing Inotify TorrentParser MagicMime Servant Benchmark Feeds Testsuite S3 WebDAV
dependency versions: aws-0.24.4 bloomfilter-2.0.1.2 crypton-1.0.4 DAV-1.3.4 feed-1.3.2.1 ghc-9.10.1 http-client-0.7.19 persistent
-sqlite-2.13.3.0 torrent-10000.1.3 uuid-1.3.16 yesod-1.6.2.1
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E S
HA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2
B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S
224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL GITBUNDLE GITMANIFEST VURL X*
remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs httpalso borg rclone hoo
k external compute mask
operating system: linux x86_64
supported repository versions: 8 9 10
upgrade supported from repository versions: 0 1 2 3 4 5 6 7 8 9 10

Please provide any additional information below.

# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log


# End of transcript or log.

Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)

RSS Atom

comment 1

The failures are mostly of two varieties.

type A:

  add:                            FAIL (2.20s)
    ./Test/Framework.hs:395:
    checkcontent foo
    expected: "annexed file content"
     but got: "could not read file"

type B:

  init:              OK (1.98s)
  add:               FAIL (5.90s)
    ./Test/Framework.hs:92:
    unlock failed with unexpected exit code (transcript follows)
unlock sha1foo cp: cannot open '.git/annex/objects/3j/xV/SHA1-s25--ee80d2cec57a3810db83b80e1b320df3a3721ffa/SHA1-s25--ee80d2cec57a3810db83b80e1b320df3a3721ffa' for reading: No such file or directory

In both cases, a git-annex add is succeeding, but the annex objects directory is somehow not getting populated. Or at least, a subsequent read of a file in it has the filesystem not knowing the file that the add put there is there.

It seems quite likely a lot of other tests would also fail, but they are being skipped because the add tests fail.

In one case, the add tests are succeeding (on an adjusted unlocked branch), but then a subsequent test fails:

git-remote-annex exporttree:      FAIL (8.45s)
  ./Test/Framework.hs:92:
  export failed with unexpected exit code (transcript follows)
  mv: cannot move '.git/annex/othertmp/89ddefa4-a04c-11.0/89ddefa4-a04c-11' to '.git/annex/export.ex/89ddefa4-a04c-11ef-818d8a-1-c6223d6': Device or resource busy
  mv: cannot move '.git/annex/othertmp/89ddefa4-a04c-11.0/89ddefa4-a04c-11' to '.git/annex/export.ex/89ddefa4-a04c-11ef-818d8a-2-c718026': Device or resource busy
  git-annex: renamePath:rename '.git/annex/othertmp/89ddefa4-a04c-11.0/89ddefa4-a04c-11' to '.git/annex/export.ex/89ddefa4-a04c-11ef-87b5-e880882a4f98': resource busy (Device or resource busy)

Comment by joey — Fri Aug 29 15:16:00 2025

comment 2

I'm not familiar with beegfs, but its documentation such as this https://doc.beegfs.io/latest/architecture/overview.html makes me wonder if it manages to behave consistently as we would expect a filesystem to behave.

In particular, we have a file being moved from one directory to another directory. Beegfs's docs says it will pick a random metadata node for each directory. So there can be two metadata nodes that have to be updated for a move. If one node somehow lags seeing the update, could that result in the file not appearing as present in the destination directory after the move?

I'm only speculating about how beegfs might work, but it seems unlikely that git-annex has a bug that causes it to lose an annexed file when all it's done is move it to the objects directory, that only manifests on this one filesystem.

A good next step might be to try manually adding an annexed file and see if there is some lag between git-annex add and being able to read the content of the symlink. Eg, compare:

echo foo > foo
git-annex add foo; cat foo
echo bar > bar
git-annex add bar; sleep 1m; cat bar

Comment by joey — Fri Aug 29 15:31:42 2025

odd odd filesystem

eh, it is indeed quite a f...un filesystem: even chmod might endup unhappy

chmod +w -R test-repo
chmod: changing permissions of 'test-repo/.git/annex/objects/91/9x/SHA256E-s4--b5bb9d8014a0f9b1d61e21e796d78dccdf1352f23cd32812f4850b878ae4944c/SHA256E-s4--b5bb9d8014a0f9b1d61e21e796d78dccdf1352f23cd32812f4850b878ae4944c': No such file or directory
chmod: changing permissions of 'test-repo/.git/annex/objects/g7/9v/SHA256E-s4--7d865e959b2466918c9863afca942d0fb89d7c9ac0c99bafc3749504ded97730/SHA256
E-s4--7d865e959b2466918c9863afca942d0fb89d7c9ac0c99bafc3749504ded97730': No such file or directory

and it might indeed take minute(s) for filesytem to become "consistent" with expected state. And it seems that even a minute might not be enough!!!

ok
(recording state in git...)
(test_script.sh:14):
sleep 1m
(test_script.sh:14):
cat bar
cat: bar: No such file or directory

with this full tune up script

#!/bin/bash

if [ -e test-repo ]; then 
    chmod +w -R test-repo; 
    rm -rf test-repo;  
fi
mkdir test-repo; cd test-repo; git init; git annex init;

echo foo > foo
git-annex add foo; cat foo
echo bar > bar
# git-annex add bar; sleep 1m; cat bar
git-annex add bar
start_time=$(date +%s)
while ! cat bar 2>/dev/null; do
    sleep 1
    echo -n .
done
end_time=$(date +%s)
wait_time=$((end_time - start_time))
cat bar
echo "Waited ${wait_time} seconds for bar to appear"

it Waited 450 seconds for bar to appear... and on this funny system bash would report that file is available so symlink would not be broken to those tests:

$> test -e bar || echo fail
$> cat bar
cat: bar: No such file or directory

I will report to sysadmins -- may be they have ideas/feedback ;-)

Comment by yarikoptic — Tue Sep 2 15:06:43 2025

Add a comment