Tests v8 locked: rsync remote: FAIL

Please describe the problem.

2022-04-28T03:40:48.4460611Z   Repo Tests v8 locked
2022-04-28T03:40:48.4460898Z     Init Tests
2022-04-28T03:40:48.4461239Z       init:                                               OK (0.08s)
2022-04-28T03:40:48.4461605Z       add:                                                OK (0.31s)
2022-04-28T03:40:48.4461985Z     preferred content:                                    OK (1.26s)
2022-04-28T03:40:48.4462414Z     rsync remote:                                         FAIL (0.63s)

across setups:

(git)smaug:/mnt/datasets/datalad/ci/git-annex/builds/2022/04[master]cron-20220428
$> git grep -n FAIL| grep test-annex                                                                
build-macos.yaml-670-0346631d-failed/1_test-annex (normal, macos-10.15).txt:1580:2022-04-28T02:40:19.9278580Z     rsync remote:                                 FAIL (1.44s)
build-macos.yaml-670-0346631d-failed/2_test-annex (crippled-tmp, macos-10.15).txt:1692:2022-04-28T02:44:48.5060190Z     rsync remote:                                 FAIL (1.41s)
build-macos.yaml-670-0346631d-failed/3_test-annex (normal, macos-latest).txt:1581:2022-04-28T02:40:39.2774670Z     rsync remote:                                 FAIL (1.32s)
build-macos.yaml-670-0346631d-failed/test-annex (crippled-tmp, macos-10.15)/8_Run tests.txt:1425:2022-04-28T02:44:48.5060140Z     rsync remote:                                 FAIL (1.41s)
build-macos.yaml-670-0346631d-failed/test-annex (normal, macos-10.15)/8_Run tests.txt:1314:2022-04-28T02:40:19.9278580Z     rsync remote:                                 FAIL (1.44s)
build-macos.yaml-670-0346631d-failed/test-annex (normal, macos-latest)/8_Run tests.txt:1314:2022-04-28T02:40:39.2774650Z     rsync remote:                                 FAIL (1.32s)
build-ubuntu.yaml-675-0346631d-failed/1_test-annex (normal, ubuntu-latest).txt:1427:2022-04-28T03:40:48.4462418Z     rsync remote:                                         FAIL (0.63s)
build-ubuntu.yaml-675-0346631d-failed/2_test-annex (crippled-tmp, ubuntu-latest).txt:1577:2022-04-28T03:41:44.8645593Z     rsync remote:                                         FAIL (0.64s)
build-ubuntu.yaml-675-0346631d-failed/4_test-annex (nfs-home, ubuntu-latest).txt:1513:2022-04-28T03:45:52.6285760Z     rsync remote:                                         FAIL (1.56s)
build-ubuntu.yaml-675-0346631d-failed/test-annex (crippled-tmp, ubuntu-latest)/7_Run tests.txt:1334:2022-04-28T03:41:44.8645589Z     rsync remote:                                         FAIL (0.64s)
build-ubuntu.yaml-675-0346631d-failed/test-annex (nfs-home, ubuntu-latest)/7_Run tests.txt:1270:2022-04-28T03:45:52.6285755Z     rsync remote:                                         FAIL (1.56s)
build-ubuntu.yaml-675-0346631d-failed/test-annex (normal, ubuntu-latest)/7_Run tests.txt:1185:2022-04-28T03:40:48.4462414Z     rsync remote:                                         FAIL (0.63s)

and for some time already (as pinged in an email) -- since 20220420

$> git grep -n rsync.*FAIL| grep test-annex | grep ubuntu-late
cron-20220420/build-ubuntu.yaml-667-0346631d-failed/1_test-annex (normal, ubuntu-latest).txt:1427:2022-04-20T03:30:39.0198567Z     rsync remote:                                         FAIL (0.70s)
cron-20220420/build-ubuntu.yaml-667-0346631d-failed/2_test-annex (crippled-tmp, ubuntu-latest).txt:1582:2022-04-20T03:31:17.9571887Z     rsync remote:                                         FAIL (0.62s)
cron-20220420/build-ubuntu.yaml-667-0346631d-failed/4_test-annex (nfs-home, ubuntu-latest).txt:1511:2022-04-20T03:34:57.9524586Z     rsync remote:                                         FAIL (1.57s)
...

fixed --Joey

RSS Atom

comment 1

The failure message is:

  ./Test/Framework.hs:328:
  able to modify annexed file's foo content

Apparently some change to rsync in 3.2.4 has messed up something involving permissions.

See also another bug caused by the new rsync http://bugs.debian.org/1010397

Comment by joey — Mon May 2 16:53:13 2022

Remove comment

comment 2

Here is the bug in action:

-r--r--r-- 1 joey joey 30 May  2 12:42 .git/annex/objects/Gj/8J/SHA256E-s30--3d65cafd9435fde3867a527d75ff8aea05a3632cb60574d45e0fc277f06c8a64/SHA256E-s30--3d65cafd9435fde3867a527d75ff8aea05a3632cb60574d45e0fc277f06c8a64
joey@darkstar:/tmp/x>git-annex copy --to foo
copy x (to foo...) 
ok
joey@darkstar:/tmp/x>ls -l .git/annex/objects/Gj/8J/SHA256E-s30--3d65cafd9435fde3867a527d75ff8aea05a3632cb60574d45e0fc277f06c8a64/SHA256E-s30--3d65cafd9435fde3867a527d75ff8aea05a3632cb60574d45e0fc277f06c8a64
-rwxr--r-- 1 joey joey 30 May  2 12:42 .git/annex/objects/Gj/8J/SHA256E-s30--3d65cafd9435fde3867a527d75ff8aea05a3632cb60574d45e0fc277f06c8a64/SHA256E-s30--3d65cafd9435fde3867a527d75ff8aea05a3632cb60574d45e0fc277f06c8a64*

At first I thought this was rsync modifying the permissions of the source file.

But no... 17b20a24502aee3bfc5683146c3899a233295aea changed how temp directories get cleaned up. removePathForcibly is actually changing the permissions of the object file hard link in the rsynctmp directory when deleting that directory. Which also changes the permissions of the object file.

Filed a bug on removePathForcibly. https://github.com/haskell/directory/issues/135

I think this makes removePathForcibly unsuitable for general purpose use in git-annex, because there are just too many ways for a hard link to enter the picture. (Eg annex.thin, or even a user making their own hard link that git-annex does not know about.)

So, I've reverted that commit, and put in a more targeted fix for the problem it was addressing.

Comment by joey — Mon May 2 17:08:15 2022

Remove comment

Add a comment