projects/datalad/bugs-done/1 test failure under conda on Windows 10: prop_view_roundtripsyohhttp://git-annex.branchable.com/projects/datalad/bugs-done/1_test_failure_under_conda_on_Windows_10__58___prop__95__view__95__roundtrips/git-annexikiwiki2023-01-05T17:30:31Zcomment 1http://git-annex.branchable.com/projects/datalad/bugs-done/1_test_failure_under_conda_on_Windows_10__58___prop__95__view__95__roundtrips/comment_1_3a2be0a1cd74c7dba3b07e1300ce2a30/joey2023-01-05T17:30:31Z2020-10-26T16:03:40Z
<p>I can reproduce it in a windows VM running
<code>git-annex test --quickcheck-replay=742853</code></p>
<p>These quickcheck tests test random input so not flaky exactly.</p>
<p>Does not happen with that seed on linux, so it probably involves something
encoding specific. An area where the windows port is known to have
extensive problems.</p>
<p>(<a href="http://source.git-annex.branchable.com/?p=source.git;a=commitdiff;h=1b8026b2cbc8df0274082c5f08a8b4f8ca47c5c9">1b8026b2cbc8df0274082c5f08a8b4f8ca47c5c9</a> was similar,
although that was MetaField and this appears to be MetaValue.)</p>
comment 2http://git-annex.branchable.com/projects/datalad/bugs-done/1_test_failure_under_conda_on_Windows_10__58___prop__95__view__95__roundtrips/comment_2_1945040bcdf209b9a0faecd4a77e0f35/yarikoptic2023-01-05T17:30:31Z2020-10-26T17:57:12Z
you say "windows port", I say "windows as a whole", e.g. today revelation (or just a come back if I ran into it before but forgot) to me <a href="https://github.com/datalad/datalad/issues/5097">was inability to have a file/directory named <code>con</code>...</a> - no bloody sense on how such design decision has happened and how it dragged all the way into the flagman of the 2020 product.
comment 3http://git-annex.branchable.com/projects/datalad/bugs-done/1_test_failure_under_conda_on_Windows_10__58___prop__95__view__95__roundtrips/comment_3_06dd9606225cf881f7681fc27fb4833d/joey2023-01-05T17:30:31Z2020-10-26T18:41:27Z
<p>Hmm, this uses viewedFiles, which generates filenames
based on the MetaValue. Note use of pathProduct, which uses
System.FilePath.combine.</p>
<p>So, generating random ascii (including escape sequences)
bytestrings, and passing them through decodeBS to generate FilePaths,
and then operating on those filepaths. What could possibly go wrong.</p>
<p>And aha! I made pathProduct use System.FilePath.Windows.combine
and was able to reproduce the test suite failure on Linux.</p>
<p>And aha again:</p>
<pre><code>MetaValue (CurrentlySet True) "c:"
</code></pre>
<p>Which of course breaks it on windows because it wanted to generate
something like "bar/c:/baz/a" but instead it gets "c:/bar/baz/a"</p>
<p>git-annex does replace '/' and '\' when generating these filenames.
Not as a security measure (when the view branch is checked out, git's
security checks apply same as any branch so it piggybacks on those),
but to let the user build a view and successfully check it out
when their metadata happens to include such stuff.</p>
<p>However, windows does have enough special filenames and gotchas
that it simply does not seem to make sense for git-annex to try to work
around them all in the view code. If a MetaValue happens to end with a
period, or is "nul", and so the generated filename is illegal on Windows,
it'll blow up at checkout time, and I am ok with that.</p>
<p>So I think it would make sense to also escape ':', but that's about as far
as this should go. <em>Especially</em> because the filenames it generates need to
roundtrip back to metadata cleanly, which is what this test case is
testing. While I can finesse individual characters, it would be quite hard
to make a filename w/o a trailing dot roundtrip back to one with it, for
example.</p>
comment 4http://git-annex.branchable.com/projects/datalad/bugs-done/1_test_failure_under_conda_on_Windows_10__58___prop__95__view__95__roundtrips/comment_4_0c6b8a3b0a6fbd700b3c8934bfec199f/yarikoptic2023-01-05T17:30:31Z2021-01-22T16:44:58Z
<p>did it come back, I see</p>
<pre><code>2021-01-22T04:32:25.5012547Z prop_view_roundtrips: FAIL (0.09s)
2021-01-22T04:32:25.5015902Z *** Failed! Falsified (after 218 tests):
2021-01-22T04:32:25.5016251Z AssociatedFile (Just "rdmBBP")
2021-01-22T04:32:25.5018130Z MetaData (fromList [(MetaField "CkL",fromList [MetaValue (CurrentlySet False) "",MetaValue (CurrentlySet True) "\SOH5:R9\EM\DC4",MetaValue (CurrentlySet True) "\STX\US\fL2\ACK|\\\r[$",MetaValue (CurrentlySet False) "\ETBRi",MetaValue (CurrentlySet False) "/\FS}",MetaValue (CurrentlySet True) "W",MetaValue (CurrentlySet False) "X=sQh\NAK^",MetaValue (CurrentlySet False) "l\SUB\a"]),(MetaField "jM",fromList [MetaValue (CurrentlySet False) "",MetaValue (CurrentlySet False) "\FSSivk",MetaValue (CurrentlySet True) "J'<\SYN\STXGJP"]),(MetaField "V",fromList [MetaValue (CurrentlySet False) "",MetaValue (CurrentlySet True) "\n\NUL",MetaValue (CurrentlySet True) "\r",MetaValue (CurrentlySet False) "+X",MetaValue (CurrentlySet True) "@aN\t~c\SIy",MetaValue (CurrentlySet False) "K>xq",MetaValue (CurrentlySet True) "a:"]),(MetaField "W",fromList [MetaValue (CurrentlySet True) "0\DC4qL",MetaValue (CurrentlySet False) "K",MetaValue (CurrentlySet False) "LD\DC3<M",MetaValue (CurrentlySet False) "a\v",MetaValue (CurrentlySet True) "dO",MetaValue (CurrentlySet True) "w\EOT"])])
2021-01-22T04:32:25.5020545Z True
2021-01-22T04:32:25.5020894Z Use --quickcheck-replay=455629 to reproduce.
</code></pre>
<p>on https://github.com/datalad/git-annex/runs/1746587663?check_suite_focus=true with <code>8.20201129+git169-gaa07e68ed_x64</code></p>
comment 4http://git-annex.branchable.com/projects/datalad/bugs-done/1_test_failure_under_conda_on_Windows_10__58___prop__95__view__95__roundtrips/comment_4_c2f248745388b508337c5097baaa1979/joey2023-01-05T17:30:31Z2021-01-22T16:59:55Z
<p>Not sure if this is really the same bug, though certainly related. These
quickcheck tests are fuzz tests, they can find numerous bugs, that's kind of
the point of them. In any case, posting to a closed bug report risks your
followup being lost and deprioritises it.</p>
<p>The problem this new failure shows is that toViewPath is failing to escape the
final character in the path in some cases. Which is not a windows-specific
bug at all really, it could also happen with a metadata value such as "foo/"
being set on linux. Fixed that bug.</p>
<p>Which shows the point of these quickcheck fuzz tests: To be able to catch
lots of different bugs with a single test case.</p>