Please describe the problem.
$> git-annex version | head -n 1
git-annex version: 10.20260316-gf01ba218ffb36e8607516d9895dfaeaeaf101a05
(git)smaug:/tmp/ds000113[master]git
$> time git-annex find --not --in here --fast | head -n 1
derivatives/linear_anatomical_alignment/sub-01/ses-forrestgump/func/sub-01_ses-forrestgump_task-forrestgump_rec-XFMdico7Tad2grpbold7Tad_run-01_bold.mat
is hanging although IMHO should have exited right after spitting out first entry. Overall it takes under second to list them
$> time git-annex find --not --in here --fast > /dev/null
git-annex find --not --in here --fast > /dev/null 0.45s user 0.09s system 132% cpu 0.408 total
but if piped -- stall!
it did manage to quit once
$> git annex find --not --in here --fast | head -n 1
derivatives/linear_anatomical_alignment/sub-01/ses-forrestgump/func/sub-01_ses-forrestgump_task-forrestgump_rec-XFMdico7Tad2grpbold7Tad_run-01_bold.mat
git-annex: <stdout>: hFlush: resource vanished (Broken pipe)
with system wide installed 10.20250416 but I failed to reproduce
What steps will reproduce the problem?
FWIW -- above on https://github.com/OpenNeuroDatasets/ds000113.git
I didn't check other commands , but they all should behave sane as not to stall if piped ! Would be great if there was a test to ensure that long term.
well -- head also stalls
workaround
I was able to reproduce this, but only when I ran git-annex with FD 2 closed.
I'm curious if you also ran git-annex with FD 2 closed?
I suspect what must be happening is that the exception handler crashes when outputting to stderr, which causes an exception to be thrown, leading to a crash loop.
If so, it would seem likely to be a bug in ghc/base. And I'd not be surprised to find such a bug somewhere in its bug tracker. In fact, I almost remember finding this same behavior before, which may have helped me guess this due was FD 2 being closed.
With that said, I've not been able to reproduce the behavior yet with a simpler haskell program like this one:
But, that simple program also doesn't throw "hFlush: resource vanished (Broken pipe)" when piped to
head -n1, so it's not quite replicating what git-annex does.Turns out that a git-annex that does not use sanitizeTopLevelExceptionMessages will not have this behavior.
Without that,
git-annex find | head -n1does not display the broken pipe exception, and so the crash loop never happens when stderr is closed.That is relatively new; it was added in git-annex 10.20230407.
And apparently what ghc's default exception display mechanism does is actually to avoid displaying anything for the broken pipe exception for stdout. Which leads to the more typical unix behavior of silently stopping on SIGPIPE.
See also https://mail.haskell.org/pipermail/haskell-cafe/2023-May/136194.html
how do I discover that ? it worked fine if I redirect stderr:
and damn it -- after that it started to work in that shell:
/mnt/btrfs/datasets/datalad/crawl/dandi/dandisets/000026-- still running (likely more stuff spit out to stdout), even withI will keep it going to see if ever returns, will check in an hour. FWIW -- that dataset is here
git clone https://github.com/dandisets/000026.git && cd 000026 && git annex init && git annex find --not --in here --fast | head -n 1-- stalled as well. I wonder if replicates for you. Also there with2>/dev/nullreturned right away, but on rerun without - stalled again.