projects/datalad/bugs-done/v6 - under subdir: git add "whines", git commit "blows"yohhttp://git-annex.branchable.com/projects/datalad/bugs-done/v6_-_under_subdir__58___git_add___34__whines__34____44___git_commit___34__blows__34__/git-annexikiwiki2023-01-05T17:30:31Zthe reason is system wide git version!http://git-annex.branchable.com/projects/datalad/bugs-done/v6_-_under_subdir__58___git_add___34__whines__34____44___git_commit___34__blows__34__/comment_1_1c86937cee067b5980852fcb066b0bfb/yarikoptic2023-01-05T17:30:31Z2018-09-07T00:13:33Z
<p>HA -- it is compatibility issue with git somewhere.<br />
I had system wide 1:2.13.0~rc1+next.20170430-1~nd90+1 installation of git, and that was the reason. Similar blows with 1:2.11.0-3+nd1~nd90+1, but works fine with 1:2.19.0~rc2-1 .
At least that explains how we did not run into it while testing in DataLad:</p>
<ol>
<li>we use git bundled with git-annex when git annex standalone is installed</li>
<li>We use "git annex add" and not "git add" directly even in v6 mode repos</li>
</ol>
<p>But it brings the point of possibly needing to test with a regular debian build of git annex more which would use system-wide git. Also that may be git annex should test git version used and blow with informative message whenever git is too old?</p>
comment 2http://git-annex.branchable.com/projects/datalad/bugs-done/v6_-_under_subdir__58___git_add___34__whines__34____44___git_commit___34__blows__34__/comment_2_bd552525b7bb2ceaad72c8fa3bf41c6b/joey2023-01-05T17:30:31Z2018-09-11T17:39:05Z
<p>There are a few places in git-annex where it runs git with the cwd
overridden, eg to the top of a git repository.
If it somehow comes up with a path that doesn't exist,
that would explain this error. The most likely one is
in <code>Git.Config.read'</code>, which git-annex always runs at startup.</p>
<p>So it probably comes down to git running the clean filter with
the wrong <code>GIT_DIR</code> setting or something like that, leading to this as a
cascading failure. And I vaguely remember seeing something like that in the
git commit log, or having dealt with such a problem before.</p>
<p>We could make git-annex upgrade fail to run with too old a git (and it
probably should at least when used with a git too old to support
smudge/clean filters at all!), but it wouldn't help if the repository was
upgraded in a different environment. git is going to run the git-annex via
the clean filter before any check can happen and the most git-annex can
then do is error out somehow.</p>
<p>See also
<a href="http://git-annex.branchable.com/forum/Why_git_commit_fails_from_within_a_newly_git-annexed_subdir__63__/">http://git-annex.branchable.com/forum/Why_git_commit_fails_from_within_a_newly_git-annexed_subdir__63__/</a>
which seems to be the same problem without v6; they didn't say what git
version.</p>
comment 3http://git-annex.branchable.com/projects/datalad/bugs-done/v6_-_under_subdir__58___git_add___34__whines__34____44___git_commit___34__blows__34__/comment_3_8cfa8fe790ff2586374f46ed10cc8fbe/joey2023-01-05T17:30:31Z2018-09-11T18:08:43Z
<p>Ah yes, I was remembering this workaround in git-annex:</p>
<pre><code>* v6: Work around git bug that runs smudge/clean filters at the top of the
repository while passing them a relative GIT_WORK_TREE that may point
outside of the repository, by using GIT_PREFIX to get back to the
subdirectory where a relative GIT_WORK_TREE is valid.
</code></pre>
<p>Perhaps that broke with some older versions of git.
<a href="http://source.git-annex.branchable.com/?p=source.git;a=commitdiff;h=e50ed4ba48f93cf0addb3638a4a9605a10f17976">e50ed4ba48f93cf0addb3638a4a9605a10f17976</a>
has the gory details, which includes a git bug.</p>
<p>A good way to debug this is to set:</p>
<pre><code>git config filter.annex.clean 'bash -c "set | grep GIT_ >&2; pwd >&2; git-annex smudge --clean %f"'
</code></pre>
<p>Then when git runs the clean filter it will display the git environment variables.</p>
comment 4http://git-annex.branchable.com/projects/datalad/bugs-done/v6_-_under_subdir__58___git_add___34__whines__34____44___git_commit___34__blows__34__/comment_4_40e9a5b641200c92725df31a654b6a05/joey2023-01-05T17:30:31Z2018-09-11T18:33:22Z
<p>Reproduced in a debian stable chroot with current git-annex in it.</p>
<pre><code>root@darkstar:/tmp# mkdir repo; cd repo; git init; git annex init; git annex upgrade; git config filter.annex.clean 'bash -c "set | grep GIT_ >&2; pwd >&2; git-annex smudge --clean %f"' ; mkdir -p subdir; cd subdir; touch file; git add file
...
GIT_DIR=.git
GIT_PREFIX=subdir/
/tmp/repo
git-annex: git: createProcess: runInteractiveProcess: chdir: does not exist (No such file or directory)
</code></pre>
<p>Verifies my theory.</p>
comment 5http://git-annex.branchable.com/projects/datalad/bugs-done/v6_-_under_subdir__58___git_add___34__whines__34____44___git_commit___34__blows__34__/comment_5_eaafc970cce2c71ce29f136c90e04504/joey2023-01-05T17:30:31Z2018-09-11T18:51:07Z
<p>Unfortunately git 2.11.0 has the bug that I made git-annex look at
<code>GIT_PREFIX</code> to avoid.</p>
<pre><code>root@darkstar:/tmp/repo/foo/bar# git --work-tree=../.. ls-files --modified /tmp/repo/
BASH_EXECUTION_STRING='set | grep GIT_ >&2; pwd >&2; git-annex smudge --clean '\''subdir/file'\'''
GIT_DIR=/tmp/repo/.git
GIT_PREFIX=foo/bar/
GIT_WORK_TREE=../..
/tmp/repo
</code></pre>
<p>So, it's not as simple as not looking at <code>GIT_PREFIX</code> at all with this
version of git. When <code>GIT_WORK_TREE</code> is relative, and <code>GIT_PREFIX</code> is set,
it needs to combine them to get the actual path to the work tree.</p>
<p>Perhaps git-annex should only use <code>GIT_PREFIX</code>
for fixup of relative <code>GIT_WORK_TREE</code>, but not for <code>GIT_DIR</code>. I've so
far only seen it be needed for <code>GIT_WORK_TREE</code>; I only made it also
be used for <code>GIT_DIR</code> out of an assumption git would be consistent in its
bugginess.</p>
<p>Oh hell, here's another way it fails, with git 2.19:</p>
<pre><code>joey@darkstar:/tmp/repo/subdir/foo> git --work-tree=../.. status
BASH_EXECUTION_STRING='set | grep GIT_ >&2; pwd >&2; git-annex smudge --clean '\''subdir/file'\'''
GIT_DIR=/tmp/repo/.git
GIT_EXEC_PATH=/usr/lib/git-core
GIT_MERGE_AUTOEDIT=no
GIT_PAGER=
GIT_PREFIX=subdir/foo/
GIT_WORK_TREE=.
/tmp/repo
git-annex: subdir/file: getFileStatus: does not exist (No such file or directory)
git-annex: smudge: 1 failed
</code></pre>
<p>That one does not involve <code>GIT_DIR</code>, instead <code>GIT_WORK_TREE</code> is not relative to
<code>GIT_PREFIX</code> so the workaround that assumes it is breaks. I guess that we
could say that <code>GIT_DIR</code> is not relative to some directory in this case,
so the <code>GIT_PREFIX</code> is irrelevant.</p>
<p>Kind of feels like git's behavior is so inconsistent and ill-specified
that trying to work around the bugs in it is likely to just be a neverending
source of bugs.</p>
<p>Unfortunately the git devs have so far ignored my bug report despite it
having an easy test case needing nothing more than a simple shell script to
reproduce. And there are plenty of differently broken git versions out there
even if they eventually fix it, so it seems git-annex has to deal with this
mess somehow.</p>
<p>I'm going with not using <code>GIT_PREFIX</code> for <code>GIT_DIR</code> and not for
<code>GIT_WORK_TREE=.</code> and hope that's enough.</p>