Recent comments posted to this site:

what kind of a "fault" could it be? ;)
Comment by yarikoptic Mon Dec 5 18:08:02 2022

The rerun did not get stuck. So it's not something directly the fault of the script running git-annex test, I think.

Comment by joey Mon Dec 5 17:48:52 2022

https://github.com/yesodweb/persistent/issues/1441 is having some useful discussion. There are changes that would speed it up 2x. The best path may be improving sqlite to optimise insert_.

Comment by joey Mon Dec 5 17:29:11 2022

Not just limited to numcopies:

{"command":"fsck","success":true,"key":"SHA256E-s119046--239da5a85ddf8c4071d8803a864a896d13e2a2fd65fd5684fc2f6dcaf264e875.jpg","error-messages":[],"file":"12/03/chandrian_22:37:41.jpg","note":"checksum...","input ":["12/03/chandrian_22:37:41.jpg"]} Based on the location log, 12/03/chandrian_23:05:41.jpg
was expected to be present, but its content is missing.

Comment by kanak Sun Dec 4 17:40:38 2022
Thank you! That is simply perfect.
Comment by agschaid Thu Dec 1 21:12:11 2022
git annex adjust --hide-missing
Comment by Lukey Thu Dec 1 20:49:35 2022

I have re-ran the command to see if the bug replicates..

note: it was considerable amount of time (days?) for it to take there ;) I made a copy test_fs_testonly.py where I removed all other "benchmarks" prior running annex test -- might get there faster if works so feel free to interrupt and rerun that one. That code is old (circa 2014 ;)) , and I should do some face lift but haven't had a chance yet :-/

Comment by yarikoptic Tue Nov 29 22:25:42 2022

I examined this situation on the machine. I had to ctrl-z the process to get a shell.

Then I checked what the stdin of 3710605 was, and it was still open, and was a pipe from 3710526. So that's why the p2pstdio process was still running; that pipe should be closed when the parent git-annex is done, but the parent was apparently stuck on something.

Then I straced 3710605 but it was suspended (oops). So I ran fg. This somehow unstuck everything! The test suite finished up very fast.

Here is what it output for the test that had gotten stuck:

Tests
  Repo Tests v8 unlocked
    Init Tests
      init:            OK (2.54s)
      add:             OK (5.48s)
    move (ssh remote): FAIL (958561.16s)
      ./Test/Framework.hs:398:
      bad content in location log for foo key SHA256E-s20--e394a389d787383843decc5d3d99b6d184ffa5fddeec23b911f9ee7fc8b9ea77 uuid UUID "c129397d-8209-40ea-8347-16a8c3fe69de"
      expected: True
       but got: False

That seems like it must come from here in the test suite:

    git_annex "move" ["--to", "origin", annexedfile] "move --to of file"
    inmainrepo $ annexed_present annexedfile

So it seems that the content was moved back to origin successfully (annexed_present checks that the object file is present before checking the location log), but that the location log didn't get updated. Need to check if that update would have been done by the p2pstdio process or the move process.

Why would a SIGCONT have unstuck it I wonder?

I have re-ran the command to see if the bug replicates..

Comment by joey Tue Nov 29 21:45:15 2022
Thank you Joey - I wrote my previous comment as your answer was coming in. Much obliged!
Comment by cnjr2 Tue Nov 29 16:53:39 2022
I have searched around some more and found a related bug report. As per your comment what constitutes a file extension is determined by heuristics. So I guess my question would become: Would it be a bad idea to have long (and ugly) file extensions (i.e. .numbers) be recognized as as extensions? What would be situations where that would go wrong?
Comment by cnjr2 Tue Nov 29 16:51:44 2022