1 test failure under conda on Windows 10: prop_view

Please describe the problem.

What steps will reproduce the problem?

I am plowing through on making git-annex available within conda-forge "natively" for Windows. For now I just took the recently built installer, the one now available from datasets.datalad.org and built on datalad-extensions github setup. I just extracted git-annex component from the installer and placed them within conda hierarchy (installed posix package with all the needed basic tools. Overall -- looks great, but:

    prop_view_roundtrips:                                 FAIL (0.30s)
      *** Failed! Falsified (after 524 tests and 1 shrink):
      "a"
      MetaData (fromList [(MetaField "8",fromList [MetaValue (CurrentlySet False) "",MetaValue (CurrentlySet True) "\nD\EM",MetaValue (CurrentlySet True) "GO`!)",MetaValue (CurrentlySet False) "k\FS\CAN"]),(MetaField "dU",fromList [MetaValue (CurrentlySet True) "",MetaValue (CurrentlySet False) "\NUL44Vfm[\t",MetaValue (CurrentlySet True) "\nLMEgYc",MetaValue (CurrentlySet True) "\SO[",MetaValue (CurrentlySet True) "\FS\DC4\DLE\"3",MetaValue (CurrentlySet True) ";\f0&Wc\GS{^",MetaValue (CurrentlySet True) "D",MetaValue (CurrentlySet True) "c:"]),(MetaField "sV",fromList [MetaValue (CurrentlySet True) "",MetaValue (CurrentlySet False) "\STX8#w",MetaValue (CurrentlySet False) "\ny",MetaValue (CurrentlySet False) "\DC4qOq",MetaValue (CurrentlySet True) "\FSbqjq",MetaValue (CurrentlySet True) "T_bx%[lN",MetaValue (CurrentlySet True) "W0`",MetaValue (CurrentlySet True) "~ ueY"]),(MetaField "V",fromList [MetaValue (CurrentlySet False) "",MetaValue (CurrentlySet False) "\t\DC1~`\SOHv\DC1",MetaValue (CurrentlySet True) "\DLE3",MetaValue (CurrentlySet True) "/MZh$",MetaValue (CurrentlySet False) "0",MetaValue (CurrentlySet False) "MEulc",MetaValue (CurrentlySet True) "P5D",MetaValue (CurrentlySet True) "i|S,",MetaValue (CurrentlySet True) "x|C"])])
      True
      Use --quickcheck-replay=742853 to reproduce.

unfortunately I cannot tell from that output what could be the problem. Please let me know if hard to figure it out and I should provide access to such environment (ATM needs effort, so I do not want to spend time on that unless "no other way")

And it seems it might be a flaky test -- I started another run, it is still running but I this test did not fail

$ grep prop_view_roundtrips git-annex-test-miniconda*.log
git-annex-test-miniconda-2.log:    prop_view_roundtrips:                                 OK (2.51s)
git-annex-test-miniconda.log:    prop_view_roundtrips:                                 FAIL (0.30s)

Cheers,

fixed --Joey

RSS Atom

comment 1

I can reproduce it in a windows VM running git-annex test --quickcheck-replay=742853

These quickcheck tests test random input so not flaky exactly.

Does not happen with that seed on linux, so it probably involves something encoding specific. An area where the windows port is known to have extensive problems.

(1b8026b2cbc8df0274082c5f08a8b4f8ca47c5c9 was similar, although that was MetaField and this appears to be MetaValue.)

Comment by joey — Mon Oct 26 16:03:40 2020

Remove comment

comment 2

you say "windows port", I say "windows as a whole", e.g. today revelation (or just a come back if I ran into it before but forgot) to me was inability to have a file/directory named con... - no bloody sense on how such design decision has happened and how it dragged all the way into the flagman of the 2020 product.

Comment by yarikoptic — Mon Oct 26 17:57:12 2020

Remove comment

comment 3

Hmm, this uses viewedFiles, which generates filenames based on the MetaValue. Note use of pathProduct, which uses System.FilePath.combine.

So, generating random ascii (including escape sequences) bytestrings, and passing them through decodeBS to generate FilePaths, and then operating on those filepaths. What could possibly go wrong.

And aha! I made pathProduct use System.FilePath.Windows.combine and was able to reproduce the test suite failure on Linux.

And aha again:

MetaValue (CurrentlySet True) "c:"

Which of course breaks it on windows because it wanted to generate something like "bar/c:/baz/a" but instead it gets "c:/bar/baz/a"

git-annex does replace '/' and '\' when generating these filenames. Not as a security measure (when the view branch is checked out, git's security checks apply same as any branch so it piggybacks on those), but to let the user build a view and successfully check it out when their metadata happens to include such stuff.

However, windows does have enough special filenames and gotchas that it simply does not seem to make sense for git-annex to try to work around them all in the view code. If a MetaValue happens to end with a period, or is "nul", and so the generated filename is illegal on Windows, it'll blow up at checkout time, and I am ok with that.

So I think it would make sense to also escape ':', but that's about as far as this should go. Especially because the filenames it generates need to roundtrip back to metadata cleanly, which is what this test case is testing. While I can finesse individual characters, it would be quite hard to make a filename w/o a trailing dot roundtrip back to one with it, for example.

Comment by joey — Mon Oct 26 18:41:27 2020

Remove comment

comment 4

did it come back, I see

2021-01-22T04:32:25.5012547Z     prop_view_roundtrips:                                 FAIL (0.09s)
2021-01-22T04:32:25.5015902Z       *** Failed! Falsified (after 218 tests):
2021-01-22T04:32:25.5016251Z       AssociatedFile (Just "rdmBBP")
2021-01-22T04:32:25.5018130Z       MetaData (fromList [(MetaField "CkL",fromList [MetaValue (CurrentlySet False) "",MetaValue (CurrentlySet True) "\SOH5:R9\EM\DC4",MetaValue (CurrentlySet True) "\STX\US\fL2\ACK|\\\r[$",MetaValue (CurrentlySet False) "\ETBRi",MetaValue (CurrentlySet False) "/\FS}",MetaValue (CurrentlySet True) "W",MetaValue (CurrentlySet False) "X=sQh\NAK^",MetaValue (CurrentlySet False) "l\SUB\a"]),(MetaField "jM",fromList [MetaValue (CurrentlySet False) "",MetaValue (CurrentlySet False) "\FSSivk",MetaValue (CurrentlySet True) "J'<\SYN\STXGJP"]),(MetaField "V",fromList [MetaValue (CurrentlySet False) "",MetaValue (CurrentlySet True) "\n\NUL",MetaValue (CurrentlySet True) "\r",MetaValue (CurrentlySet False) "+X",MetaValue (CurrentlySet True) "@aN\t~c\SIy",MetaValue (CurrentlySet False) "K>xq",MetaValue (CurrentlySet True) "a:"]),(MetaField "W",fromList [MetaValue (CurrentlySet True) "0\DC4qL",MetaValue (CurrentlySet False) "K",MetaValue (CurrentlySet False) "LD\DC3<M",MetaValue (CurrentlySet False) "a\v",MetaValue (CurrentlySet True) "dO",MetaValue (CurrentlySet True) "w\EOT"])])
2021-01-22T04:32:25.5020545Z       True
2021-01-22T04:32:25.5020894Z       Use --quickcheck-replay=455629 to reproduce.

on https://github.com/datalad/git-annex/runs/1746587663?check_suite_focus=true with 8.20201129+git169-gaa07e68ed_x64

Comment by yarikoptic — Fri Jan 22 16:44:58 2021

Remove comment

comment 4

Not sure if this is really the same bug, though certainly related. These quickcheck tests are fuzz tests, they can find numerous bugs, that's kind of the point of them. In any case, posting to a closed bug report risks your followup being lost and deprioritises it.

The problem this new failure shows is that toViewPath is failing to escape the final character in the path in some cases. Which is not a windows-specific bug at all really, it could also happen with a metadata value such as "foo/" being set on linux. Fixed that bug.

Which shows the point of these quickcheck fuzz tests: To be able to catch lots of different bugs with a single test case.

Comment by joey — Fri Jan 22 16:59:55 2021

Remove comment

Add a comment