bugsgit-annexhttp://git-annex.branchable.com/bugs/git-annexikiwiki2024-03-20T21:22:35Ztest: posix_spawnp broken 10 on darwinhttp://git-annex.branchable.com/bugs/test__58___posix__95__spawnp_broken_10_on_darwin/2024-03-20T21:22:35Z2024-03-20T21:22:35Z
<h3>Please describe the problem.</h3>
<p>We observer error in our CI:</p>
<pre><code> ./Test/Framework.hs:86:
addurl on file:///private/tmp/nix-build-git-annex-10.20231129.drv-0/git-annex-10.20231129-src/.t/25/tmprepo0/myurl failed with unexpected exit code (transcript follows)
addurl file:///private/tmp/nix-build-git-annex-10.20231129.drv-0/git-annex-10.20231129-src/.t/25/tmprepo0/myurl
git-annex: security: createProcess: posix_spawnp: does not exist (No such file or directory)
failed
addurl: 1 failed
</code></pre>
<p>https://github.com/cachix/pre-commit-hooks.nix/actions/runs/8355807260/job/22871621632</p>
<h3>What steps will reproduce the problem?</h3>
<p>build git-annex and run tests on darwin</p>
<h3>What version of git-annex are you using? On what operating system?</h3>
<p>10.20231129</p>
Get crashes when remote contains non-english charshttp://git-annex.branchable.com/bugs/Get_crashes_when_remote_contains_non-english_chars/2024-03-12T19:01:45Z2024-03-12T19:01:45Z
<p>Hi,</p>
<h3>Please describe the problem.</h3>
<p>I'm trying to set up a git-annex repo for my books/technical papers to have easy access to them on my desktop and laptop. I'm using a centralized server (following <a href="https://git-annex.branchable.com/tips/centralized_git_repository_tutorial/on_your_own_server/">this guide</a>) to make it easy to sync between my machines.</p>
<p>The issue is however that sqlite crashes when I'm trying to get a file from my server. See the log further down for the error message. I'm suspecting it is due to the repo on my server is named <code>Böcker</code> (swedish name for books). It does work if I'm cloning it locally on my server. E.g.</p>
<div class="highlight-sh"><pre class="hl">$ <span class="hl kwc">ssh</span> server
$ git clone <span class="hl opt">/</span>mnt<span class="hl opt">/</span>Valhalla<span class="hl opt">/</span>Böcker books
$ <span class="hl kwb">cd</span> books
$ git annex init
init ok
<span class="hl opt">(</span>recording state <span class="hl kwa">in</span> git...<span class="hl opt">)</span>
$ git annex get facklitteratur<span class="hl opt">/</span>rapporter<span class="hl opt">/</span><span class="hl num">143</span><span class="hl kwb">-rods</span>.pdf
get facklitteratur<span class="hl opt">/</span>rapporter<span class="hl opt">/</span><span class="hl num">143</span><span class="hl kwb">-rods</span>.pdf <span class="hl opt">(</span>from origin...<span class="hl opt">)</span>
ok
<span class="hl opt">(</span>recording state <span class="hl kwa">in</span> git...<span class="hl opt">)</span>
</pre></div>
<p>And if I use the <code>ssh://server/~/books</code> as a remote it works fine.</p>
<h3>What steps will reproduce the problem?</h3>
<div class="highlight-sh"><pre class="hl">$ <span class="hl kwc">ssh</span> server
server$ mkdir <span class="hl opt">/</span>tmp<span class="hl opt">/</span>Böcker
server$ <span class="hl kwb">cd</span> <span class="hl opt">/</span>tmp<span class="hl opt">/</span>Böcker
server$ git init
server$ git annex init
server$ <span class="hl kwc">dd</span> <span class="hl kwa">if</span><span class="hl opt">=/</span>dev<span class="hl opt">/</span>zeo of<span class="hl opt">=</span>foo bs<span class="hl opt">=</span><span class="hl num">4</span>k count<span class="hl opt">=</span><span class="hl num">1</span>
server$ git annex add foo
server$ git commit <span class="hl kwb">-m</span> <span class="hl str">"Add foo"</span>
server$ <span class="hl kwb">exit</span>
$ git clone <span class="hl kwc">ssh</span><span class="hl opt">://</span>server<span class="hl opt">/</span>tmp<span class="hl opt">/</span>Böcker <span class="hl opt">/</span>tmp<span class="hl opt">/</span>Böcker
$ <span class="hl kwb">cd</span> Böcker
$ git annex init
$ git annex get foo
</pre></div>
<p>Note that <code>foo</code> need to have some size hence the use of <code>dd</code>, just doing <code>touch foo</code> did not trigger the issue for me.</p>
<h3>What version of git-annex are you using? On what operating system?</h3>
<p>On my desktop:</p>
<div class="highlight-sh"><pre class="hl">$ git annex version
git-annex version<span class="hl opt">:</span> <span class="hl num">10.20240227</span>
build flags<span class="hl opt">:</span> Assistant Webapp Pairing Inotify DBus DesktopNotify TorrentParser MagicMime Feeds Testsuite S3 WebDAV
dependency versions<span class="hl opt">:</span> aws-0.23 bloomfilter-2.0<span class="hl num">.1.2</span> cryptonite-0.30 DAV-1.3<span class="hl num">.4</span> feed-1.3<span class="hl num">.2.1</span> ghc-9.2<span class="hl num">.5</span> http-client-0.7<span class="hl num">.13.1</span> persistent-sqlite-2.13<span class="hl num">.1.0</span> torrent-10000.1<span class="hl num">.1</span> uuid-1.3<span class="hl num">.15</span> yesod-1.6<span class="hl num">.2.1</span>
key<span class="hl opt">/</span>value backends<span class="hl opt">:</span> SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL X<span class="hl opt">*</span>
remote types<span class="hl opt">:</span> git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs httpalso borg hook external
operating system<span class="hl opt">:</span> linux x86_64
supported repository versions<span class="hl opt">:</span> <span class="hl num">8 9 10</span>
upgrade supported from repository versions<span class="hl opt">:</span> <span class="hl num">0 1 2 3 4 5 6 7 8 9 10</span>
</pre></div>
<p>Running Guix with linux kernel 6.6.18</p>
<p>On my server:</p>
<div class="highlight-sh"><pre class="hl">$ git annex version
git-annex version<span class="hl opt">:</span> <span class="hl num">10.20240227</span><span class="hl kwb">-gbee3abab14f99f3e3d981d8255ca0dd4ff124a84</span>
build flags<span class="hl opt">:</span> Assistant Webapp Pairing Inotify DBus DesktopNotify TorrentParser MagicMime Benchmark Feeds Testsuite S3 WebDAV
dependency versions<span class="hl opt">:</span> aws-0.24 bloomfilter-2.0<span class="hl num">.1.2</span> crypton-0.34 DAV-1.3<span class="hl num">.4</span> feed-1.3<span class="hl num">.2.1</span> ghc-9.2<span class="hl num">.8</span> http-client-0.7<span class="hl num">.15</span> persistent-sqlite-2.13<span class="hl num">.1.0</span> torrent-10000.1<span class="hl num">.3</span> uuid-1.3<span class="hl num">.15</span> yesod-1.6<span class="hl num">.2.1</span>
key<span class="hl opt">/</span>value backends<span class="hl opt">:</span> SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL X<span class="hl opt">*</span>
remote types<span class="hl opt">:</span> git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs httpalso borg hook external
operating system<span class="hl opt">:</span> linux x86_64
supported repository versions<span class="hl opt">:</span> <span class="hl num">8 9 10</span>
upgrade supported from repository versions<span class="hl opt">:</span> <span class="hl num">0 1 2 3 4 5 6 7 8 9 10</span>
<span class="hl kwb">local</span> repository version<span class="hl opt">:</span> <span class="hl num">10</span>
</pre></div>
<p>Running Arch Linux with kernel 6.7.9-arch1-1</p>
<h3>Please provide any additional information below.</h3>
<p>Here is the error message I get without any changes to the config (I can silence the warning by enabling annex.sshcaching).</p>
<div class="highlight-sh"><pre class="hl">⎣plattfot@desktop Böcker⎦ git annex get facklitteratur<span class="hl opt">/</span>rapporter<span class="hl opt">/</span><span class="hl num">143</span><span class="hl kwb">-rods</span>.pdf
get facklitteratur<span class="hl opt">/</span>rapporter<span class="hl opt">/</span><span class="hl num">143</span><span class="hl kwb">-rods</span>.pdf <span class="hl opt">(</span>from origin...<span class="hl opt">)</span>
You have enabled concurrency<span class="hl opt">,</span> but git-annex is not able to use <span class="hl kwc">ssh</span> connection caching. This may result <span class="hl kwa">in</span> multiple <span class="hl kwc">ssh</span> processes prompting <span class="hl kwa">for</span> passwords <span class="hl kwc">at</span> the same <span class="hl kwa">time</span>.
annex.sshcaching is not <span class="hl kwb">set</span> to true
git-annex<span class="hl opt">:</span> sqlite worker thread crashed<span class="hl opt">:</span> SQLite3 returned ErrorCan<span class="hl str">'tOpen while attempting to perform open "/mnt/Valhalla/B</span><span class="hl esc">\655</span><span class="hl str">33</span><span class="hl esc">\655</span><span class="hl str">33cker/.git/annex/keysdb/db".</span>
<span class="hl str">p2pstdio: 1 failed</span>
<span class="hl str"></span>
<span class="hl str"> Lost connection (fd:19: hGetChar: end of file)</span>
<span class="hl str"></span>
<span class="hl str"> Transfer failed</span>
<span class="hl str"></span>
<span class="hl str"> Unable to access these remotes: origin</span>
<span class="hl str"></span>
<span class="hl str"> Maybe add some of these git remotes (git remote add ...):</span>
<span class="hl str"> 8c92de49-1a89-4870-8186-b0099ad84be6 -- plattfot@server:~/books</span>
<span class="hl str">failed</span>
<span class="hl str">get: 1 failed</span>
<span class="hl str"></span>
<span class="hl str"># End of transcript or log.</span>
</pre></div>
<p>Note that this was run after I tested to create a repo with the name books.</p>
<h3>Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)</h3>
<p>Sadly no, I manage to hit this on my first try of git-annex. But I have known about git-annex for a few years, never found a good use for it in my workflow. As I've been using Syncthing for bigger files and that has been working ok. But with that you pretty much get all or nothing and conflicts are no fun to resolve. With my books I want to have my .bib file in git but somehow sync the files. I did not want to put everything in a normal git repo as the books directory takes about 64GB. After looking around I remembered git-annex and reading the walkthrough it seems to be a perfect fit! The killer feature in my opinion is that I can specify what files to sync. Which will be handy on my laptop as I don't really want to use up 64GB just so I can read one or two papers.</p>
<p>I'm not giving up on this that easily. Worst case I'll just rename my repo on my server to Books.</p>
<p>Thank you for all the hours developing this software!</p>
tls: peer does not support Extended Main Secrethttp://git-annex.branchable.com/bugs/tls__58___peer_does_not_support_Extended_Main_Secret/2024-03-07T02:33:51Z2024-03-07T02:33:51Z
<h3>Please describe the problem.</h3>
<p>On the current <a href="https://formulae.brew.sh/formula/git-annex">macOS <code>HomeBrew</code> build of the current git-annex</a> (10.20240227), it appears that the build dependencies have dragged in the <a href="https://hackage.haskell.org/package/tls-2.0.1/docs/Network-TLS.html">latest Haskell <code>tls</code>
package</a>. Which now defaults <code>supportedExtendedMainSecret</code> to <code>RequireEMS</code> (previously it seems to have been <code>AllowEMS</code>; see eg <a href="https://bugs.darcs.net/issue2715">darcs bug report of similar error</a>).</p>
<p>The result of this is that some podcast feeds, from webservers which do not support EMS, fail with an error, eg:</p>
<pre><code>importfeed https://risky.biz/feeds/risky-business
download failed: HandshakeFailed (Error_Protocol "peer does not support Extended Main Secret" HandshakeFailure)
warning: downloading the feed failed (feed: https://risky.biz/feeds/risky-business)
ok
</code></pre>
<p>(And presumably this will also affect some non-podcast HTTPS downloads; I found it in a podcast download context.)</p>
<p>I believe this "Extended Main Secret" is also known as "Extended Master Secret", aka <a href="https://www.ietf.org/rfc/rfc7627.html">RFC 7627</a>, which was written up in 2015. So I can understand why ~9 years later the Haskell <code>tls</code> library is defaulting to insisting on EMS in a new major version. Unfortunately not all webservers, especially podcast feed webservers, have caught up with this.</p>
<p>As best I can tell git annex is getting this <code>tls</code> dependency via <a href="https://hackage.haskell.org/package/http-client"><code>http-client</code></a> which uses <a href="https://hackage.haskell.org/package/http-client-tls-0.3.6.3"><code>http-client-tls</code></a>, and <code>http-client-tls</code> appears to just have a <code>tls (>=1.2)</code> dependency, which is presumably how <code>tls-2.0.0</code> / <code>tls-2.0.1</code> got dragged in, with these new defaults.</p>
<p>I'm unclear if git-annex is in a position to pass <code>AllowEMS</code> to the TLS library (and thus restore to the old default). But at least in the short term it might be worth considering doing that if possible.</p>
<h3>What steps will reproduce the problem?</h3>
<p>Currently I have three podcast feeds (two from the same webserver) which fail:</p>
<pre><code>git annex importfeed https://risky.biz/feeds/risky-business
</code></pre>
<pre><code>git annex importfeed https://risky.biz/feeds/risky-business-news
</code></pre>
<pre><code>git annex importfeed https://www.thecultureoftech.com/index.php/feed/podcast/
</code></pre>
<p>(Given the irony that the first two are are an InfoSec podcast, I have also reported this missing EMS extension support to them as well, so it may get fixed before you try it.)</p>
<p>It looks like I've also had one media file download fail repeatedly for the same reason (but the podcast feed itself downloads okay):</p>
<pre><code>git annex addurl https://traffic.omny.fm/d/clips/53b6fe2a-4ef6-4356-ae92-a61500df6da0/40b3f537-c161-4823-ae44-af3a007e121b/b2682900-b36c-447b-812d-b1290049fea8/audio.mp3
</code></pre>
<h3>What version of git-annex are you using? On what operating system?</h3>
<p>git annex 10.20240227, on macOS Ventura (13.6.3). With git annex installed from HomeBrew.</p>
<pre><code>ewen@basadi:~$ git annex version
git-annex version: 10.20240227
build flags: Assistant Webapp Pairing FsEvents TorrentParser MagicMime Benchmark Feeds Testsuite S3 WebDAV
dependency versions: aws-0.24.1 bloomfilter-2.0.1.2 crypton-0.34 DAV-1.3.4 feed-1.3.2.1 ghc-9.6.3 http-client-0.7.16 persistent-sqlite-2.13.3.0 torrent-10000.1.3 uuid-1.3.15 yesod-1.6.2.1
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL X*
remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs httpalso borg hook external
operating system: darwin x86_64
supported repository versions: 8 9 10
upgrade supported from repository versions: 0 1 2 3 4 5 6 7 8 9 10
ewen@basadi:~$
</code></pre>
<h3>Please provide any additional information below.</h3>
<div class="highlight-sh"><pre class="hl">ewen@basadi<span class="hl opt">:</span>~<span class="hl opt">/</span>Music<span class="hl opt">/</span>podcasts$ git annex importfeed https<span class="hl opt">://</span>www.thecultureoftech.com<span class="hl opt">/</span>index.php<span class="hl opt">/</span>feed<span class="hl opt">/</span>podcast<span class="hl opt">/</span>
importfeed gathering known urls ok
importfeed https<span class="hl opt">://</span>www.thecultureoftech.com<span class="hl opt">/</span>index.php<span class="hl opt">/</span>feed<span class="hl opt">/</span>podcast<span class="hl opt">/</span>
download failed<span class="hl opt">:</span> HandshakeFailed <span class="hl opt">(</span>Error_Protocol <span class="hl str">"peer does not support Extended Main Secret"</span> HandshakeFailure<span class="hl opt">)</span>
warning<span class="hl opt">:</span> downloading the feed failed <span class="hl opt">(</span>feed<span class="hl opt">:</span> https<span class="hl opt">://</span>www.thecultureoftech.com<span class="hl opt">/</span>index.php<span class="hl opt">/</span>feed<span class="hl opt">/</span>podcast<span class="hl opt">/)</span>
ok
ewen@basadi<span class="hl opt">:</span>~<span class="hl opt">/</span>Music<span class="hl opt">/</span>podcasts$
ewen@basadi<span class="hl opt">:</span>~<span class="hl opt">/</span>Music<span class="hl opt">/</span>podcasts$ git annex addurl https<span class="hl opt">://</span>traffic.omny.fm<span class="hl opt">/</span>d<span class="hl opt">/</span>clips<span class="hl opt">/</span><span class="hl num">53</span>b6fe2a-4ef6-4356-ae92-a61500df6da<span class="hl num">0</span><span class="hl opt">/</span><span class="hl num">40</span>b3f537-c161-4823-ae44-af3a007e121b<span class="hl opt">/</span>b2682900-b36c-447b-812d-b1290049fea<span class="hl num">8</span><span class="hl opt">/</span>audio.mp3
addurl https<span class="hl opt">://</span>traffic.omny.fm<span class="hl opt">/</span>d<span class="hl opt">/</span>clips<span class="hl opt">/</span><span class="hl num">53</span>b6fe2a-4ef6-4356-ae92-a61500df6da<span class="hl num">0</span><span class="hl opt">/</span><span class="hl num">40</span>b3f537-c161-4823-ae44-af3a007e121b<span class="hl opt">/</span>b2682900-b36c-447b-812d-b1290049fea<span class="hl num">8</span><span class="hl opt">/</span>audio.mp3
git-annex<span class="hl opt">:</span> HttpExceptionRequest Request <span class="hl opt">{</span>
<span class="hl kwc">host</span> <span class="hl opt">=</span> <span class="hl str">"traffic.omny.fm"</span>
port <span class="hl opt">=</span> <span class="hl num">443</span>
secure <span class="hl opt">=</span> True
requestHeaders <span class="hl opt">= [(</span><span class="hl str">"Accept-Encoding"</span><span class="hl opt">,</span><span class="hl str">""</span><span class="hl opt">),(</span><span class="hl str">"User-Agent"</span><span class="hl opt">,</span><span class="hl str">"git-annex/10.20240227"</span><span class="hl opt">)]</span>
path <span class="hl opt">=</span> <span class="hl str">"/d/clips/53b6fe2a-4ef6-4356-ae92-a61500df6da0/40b3f537-c161-4823-ae44-af3a007e121b/b2682900-b36c-447b-812d-b1290049fea8/audio.mp3"</span>
queryString <span class="hl opt">=</span> <span class="hl str">""</span>
method <span class="hl opt">=</span> <span class="hl str">"HEAD"</span>
proxy <span class="hl opt">=</span> Nothing
rawBody <span class="hl opt">=</span> False
redirectCount <span class="hl opt">=</span> <span class="hl num">10</span>
responseTimeout <span class="hl opt">=</span> ResponseTimeoutDefault
requestVersion <span class="hl opt">=</span> HTTP<span class="hl opt">/</span><span class="hl num">1.1</span>
proxySecureMode <span class="hl opt">=</span> ProxySecureWithConnect
<span class="hl opt">}</span>
<span class="hl opt">(</span>InternalException <span class="hl opt">(</span>HandshakeFailed <span class="hl opt">(</span>Error_Protocol <span class="hl str">"peer does not support Extended Main Secret"</span> HandshakeFailure<span class="hl opt">)))</span>
failed
addurl<span class="hl opt">:</span> <span class="hl num">1</span> failed
ewen@basadi<span class="hl opt">:</span>~<span class="hl opt">/</span>Music<span class="hl opt">/</span>podcasts$
</pre></div>
<h3>Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)</h3>
<p>Absolutely, I've been using git-annex as my podcatcher (among other reasons) for about a decade at this point. Thanks for developing it!</p>
unlocked files feel too slow http://git-annex.branchable.com/bugs/unlocked_files_feel_too_slow_/2024-02-19T14:21:36Z2024-02-19T04:07:04Z
<h3>Please describe the problem.</h3>
<p>I recently switched a large (~6TiB) video repository from "normal" to "unlocked" mode, mainly because some external tool was getting confused by symlinks.</p>
<p>As a result, all remotes related to that repositories have this painful, multi-hour process by which they convert their work tree to follow the new layout. That, in itself, is disturbingly slow, with little progress shown to the user. In fact, I often fear that git-annex is messing up and actually adding files as git blobs itself, which makes me interrupt it and start over again, which, obviously, makes things much worse.</p>
<p>But there are other weird things. At the moment, I'm copying files from my laptop with:</p>
<pre><code>git annex copy --from angela -J2
</code></pre>
<p>Being worried about the lack of progress, I interrupted it and ran instead:</p>
<pre><code>git annex copy --from angela -J2 --not --in here --in angela
</code></pre>
<p>That command generated zero output for 12 minutes straight. Looking at strace, it seemed to have read a large number of files in <code>.git/annex/objects</code>, which is slow.</p>
<p>Another example is, after the above command completed, i ran <code>git annex sync -g</code>, which i expected to take a handful of seconds, but the first line was a <code>commit</code> that took nearly a minute.</p>
<h3>What steps will reproduce the problem?</h3>
<p>It's not quite clear to me how to reproduce this from scratch. I guess convert an existing large repo?</p>
<h3>What version of git-annex are you using? On what operating system?</h3>
<p>10.20230802-1~bpo12+1 on debian bookworm.</p>
<h3>Please provide any additional information below.</h3>
<p>Not sure what could help here. Maybe having more feedback on what's actually happening in those minutes-long stretches would help. Git, for example, has a percentage/object count thing that pops up when things like <code>git status</code> take too long. I would love to see something similar here.</p>
<p>Also, I understand the <a href="http://git-annex.branchable.com/todo/git_smudge_clean_interface_suboptiomal/">git smudge clean interface suboptiomal</a> problem, and this might just be a specific manifestation of it. It still seems to me having some feedback, if even around the git filter boundaries, would alleviate some of those problems.</p>
<h3>Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)</h3>
<p>I use git-annex daily, and it's a lifesaver. <img src="http://git-annex.branchable.com/smileys/smile.png" alt=":)" /></p>
pull doesn't work on adjusted brancheshttp://git-annex.branchable.com/bugs/pull_doesn__39__t_work_on_adjusted_branches/2024-02-03T18:46:56Z2024-02-03T18:46:56Z
<h3>Please describe the problem.</h3>
<p>Pull works fine if both the source and destination repositories are on unadjusted branches.</p>
<p>If either of the repositories is on an adjusted branch, the pull operation does not copy anything.</p>
<h3>What steps will reproduce the problem?</h3>
<p>git annex pull --allow-unrelated-histories pull-remote</p>
<h3>What version of git-annex are you using? On what operating system?</h3>
<p>10.20231228-g5540f42e21afce6947d5410c5f18b178de6c336a</p>
<p>on Linux</p>
<h3>Please provide any additional information below.</h3>
<div class="highlight-sh"><pre class="hl"><span class="hl slc"># If you can, paste a complete transcript of the problem occurring here.</span>
<span class="hl slc"># If the problem is with the git-annex assistant, paste in .git/annex/daemon.log</span>
<span class="hl slc"># End of transcript or log.</span>
</pre></div>
<h3>Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)</h3>
<p>Yes, I am developing the easy-git-annex interface.</p>
file field omitted from import --json-progresshttp://git-annex.branchable.com/bugs/file_field_omitted_from_import_--json-progress/2024-01-30T16:42:32Z2024-01-30T16:41:55Z
<h3>Please describe the problem.</h3>
<p>When importing from a directory special remote created with importtree=yes and the --json-progress option is specified, the file field is omitted from the progress objects.</p>
<h3>What steps will reproduce the problem?</h3>
<p>The command</p>
<p>git annex import --from import-special-remote --json-progress main:Music</p>
<p>generates output with the file field omitted from the nested action object in the progress objects on line 2 and 4, below.</p>
<h3>What version of git-annex are you using? On what operating system?</h3>
<p>10.20231228-g5540f42e21afce6947d5410c5f18b178de6c336a</p>
<p>on Linux</p>
<h3>Please provide any additional information below.</h3>
<div class="highlight-sh"><pre class="hl"><span class="hl slc"># If you can, paste a complete transcript of the problem occurring here.</span>
<span class="hl slc"># If the problem is with the git-annex assistant, paste in .git/annex/daemon.log</span>
<span class="hl opt">{</span><span class="hl esc">\"</span><span class="hl kwb">command</span><span class="hl esc">\"</span><span class="hl opt">:</span><span class="hl esc">\"</span>list<span class="hl esc">\"</span><span class="hl opt">,</span><span class="hl esc">\"</span>error-messages<span class="hl esc">\"</span><span class="hl opt">:[],</span><span class="hl esc">\"</span>input<span class="hl esc">\"</span><span class="hl opt">:[],</span><span class="hl esc">\"</span>success<span class="hl esc">\"</span><span class="hl opt">:</span>true<span class="hl opt">}</span>
<span class="hl opt">{</span><span class="hl esc">\"</span>action<span class="hl esc">\"</span><span class="hl opt">:{</span><span class="hl esc">\"</span><span class="hl kwb">command</span><span class="hl esc">\"</span><span class="hl opt">:</span><span class="hl esc">\"</span>import import-special-remote<span class="hl esc">\"</span><span class="hl opt">,</span><span class="hl esc">\"</span>input<span class="hl esc">\"</span><span class="hl opt">:[]},</span><span class="hl esc">\"</span>byte-progress<span class="hl esc">\"</span><span class="hl opt">:</span><span class="hl num">9972</span><span class="hl opt">,</span><span class="hl esc">\"</span>percent-progress<span class="hl esc">\"</span><span class="hl opt">:</span><span class="hl esc">\"</span><span class="hl num">100</span><span class="hl opt">%</span><span class="hl esc">\"</span><span class="hl opt">,</span><span class="hl esc">\"</span>total-size<span class="hl esc">\"</span><span class="hl opt">:</span><span class="hl num">9972</span><span class="hl opt">}</span>
<span class="hl opt">{</span><span class="hl esc">\"</span><span class="hl kwb">command</span><span class="hl esc">\"</span><span class="hl opt">:</span><span class="hl esc">\"</span>import import-special-remote<span class="hl esc">\"</span><span class="hl opt">,</span><span class="hl esc">\"</span>error-messages<span class="hl esc">\"</span><span class="hl opt">:[],</span><span class="hl esc">\"</span>input<span class="hl esc">\"</span><span class="hl opt">:[],</span><span class="hl esc">\"</span>success<span class="hl esc">\"</span><span class="hl opt">:</span>true<span class="hl opt">}</span>
<span class="hl opt">{</span><span class="hl esc">\"</span>action<span class="hl esc">\"</span><span class="hl opt">:{</span><span class="hl esc">\"</span><span class="hl kwb">command</span><span class="hl esc">\"</span><span class="hl opt">:</span><span class="hl esc">\"</span>import import-special-remote<span class="hl esc">\"</span><span class="hl opt">,</span><span class="hl esc">\"</span>input<span class="hl esc">\"</span><span class="hl opt">:[]},</span><span class="hl esc">\"</span>byte-progress<span class="hl esc">\"</span><span class="hl opt">:</span><span class="hl num">18</span><span class="hl opt">,</span><span class="hl esc">\"</span>percent-progress<span class="hl esc">\"</span><span class="hl opt">:</span><span class="hl esc">\"</span><span class="hl num">100</span><span class="hl opt">%</span><span class="hl esc">\"</span><span class="hl opt">,</span><span class="hl esc">\"</span>total-size<span class="hl esc">\"</span><span class="hl opt">:</span><span class="hl num">18</span><span class="hl opt">}</span>
<span class="hl opt">{</span><span class="hl esc">\"</span><span class="hl kwb">command</span><span class="hl esc">\"</span><span class="hl opt">:</span><span class="hl esc">\"</span>import import-special-remote<span class="hl esc">\"</span><span class="hl opt">,</span><span class="hl esc">\"</span>error-messages<span class="hl esc">\"</span><span class="hl opt">:[],</span><span class="hl esc">\"</span>input<span class="hl esc">\"</span><span class="hl opt">:[],</span><span class="hl esc">\"</span>success<span class="hl esc">\"</span><span class="hl opt">:</span>true<span class="hl opt">}</span>
<span class="hl opt">{</span><span class="hl esc">\"</span><span class="hl kwb">command</span><span class="hl esc">\"</span><span class="hl opt">:</span><span class="hl esc">\"</span>update<span class="hl esc">\"</span><span class="hl opt">,</span><span class="hl esc">\"</span>error-messages<span class="hl esc">\"</span><span class="hl opt">:[],</span><span class="hl esc">\"</span>input<span class="hl esc">\"</span><span class="hl opt">:[],</span><span class="hl esc">\"</span>success<span class="hl esc">\"</span><span class="hl opt">:</span>true<span class="hl opt">}</span>
<span class="hl slc"># End of transcript or log.</span>
</pre></div>
<h3>Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)</h3>
<p>Yes, I am developing the easy-git-annex interface.</p>
Push to special remote includes unwanted fileshttp://git-annex.branchable.com/bugs/Push_to_special_remote_includes_unwanted_files/2024-01-31T22:19:32Z2024-01-14T16:15:35Z
<h3>Please describe the problem.</h3>
<p>When pushing to a special remote, non-largefiles (Git text files) are not filtered by the wanted setting.</p>
<h3>What steps will reproduce the problem?</h3>
<p>Push to a directory special remote with exporttree and importtree set yes. The wanted setting filters largefiles, but all Git text files are included even though some are not included in the filter.</p>
<h3>What version of git-annex are you using? On what operating system?</h3>
<p>10.20231130-g0e9bc415882a5e3a285e004cf9f80936e5762a07
Linux</p>
<h3>Please provide any additional information below.</h3>
<div class="highlight-sh"><pre class="hl"><span class="hl slc"># If you can, paste a complete transcript of the problem occurring here.</span>
<span class="hl slc"># If the problem is with the git-annex assistant, paste in .git/annex/daemon.log</span>
<span class="hl slc"># End of transcript or log.</span>
</pre></div>
<h3>Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)</h3>
identically configured remotes behave differentlyhttp://git-annex.branchable.com/bugs/identically_configured_remotes_behave_differently/2024-01-11T18:24:50Z2024-01-11T18:24:50Z
<h3>Please describe the problem.</h3>
<p>Identically configured remotes behave differently in the "git annex push" command after "git annex sync" to only one of them.</p>
<h3>What steps will reproduce the problem?</h3>
<p>I have a repository on Linux and two USB backup sticks, all Ext4 file system. At one point I accidentally did "git annex sync music-backup-one".</p>
<p>When I now "git annex push music-backup-two" it works as expected and I see the correct files on the stick and "git annex list" shows the content correctly. This is not true for "git annex push music-backup-one" as I must also issue "git annex mirror . --to=music-backup-one" for the content and "git annex list" to be correct (both commands are required in either order).</p>
<p>How do I diagnose the problem so push works without the need for mirror on music-backup-one?</p>
<p>During my investigation, I noticed the filemode (executable) bit for annexed files is not transferred to either USB drive if that is the only change (perhaps a separate issue).</p>
<h3>What version of git-annex are you using? On what operating system?</h3>
<p>10.20231130-g0e9bc415882a5e3a285e004cf9f80936e5762a07</p>
<h3>Please provide any additional information below.</h3>
<div class="highlight-sh"><pre class="hl"><span class="hl slc"># If you can, paste a complete transcript of the problem occurring here.</span>
<span class="hl slc"># If the problem is with the git-annex assistant, paste in .git/annex/daemon.log</span>
All three repos have<span class="hl opt">:</span>
<span class="hl str">'annex.largefiles'</span> <span class="hl kwb">set</span> to <span class="hl str">'mimeencoding=binary'</span><span class="hl opt">,</span>
<span class="hl str">'group'</span> <span class="hl kwb">set</span> to <span class="hl str">'manual'</span>
<span class="hl str">'wanted'</span> <span class="hl kwb">set</span> to <span class="hl str">'standard'</span> and
<span class="hl str">'numcopies'</span> <span class="hl kwb">set</span> to <span class="hl num">3</span>.
desktop-music has adjust option <span class="hl kwb">--unlock</span> and the two backups have option <span class="hl kwb">--lock</span>.
Here is the contents of .git<span class="hl opt">/</span>config <span class="hl kwa">for</span> desktop-music<span class="hl opt">:</span>
<span class="hl opt">[</span>core<span class="hl opt">]</span>
repositoryformatversion <span class="hl opt">=</span> <span class="hl num">0</span>
filemode <span class="hl opt">=</span> true
bare <span class="hl opt">=</span> false
logallrefupdates <span class="hl opt">=</span> true
<span class="hl opt">[</span>annex<span class="hl opt">]</span>
uuid <span class="hl opt">=</span> <span class="hl num">0045</span>d866-c80e-43c1-9b1b-b15a8c97c1aa
version <span class="hl opt">=</span> <span class="hl num">10</span>
adjustedbranchrefresh <span class="hl opt">=</span> true
<span class="hl opt">[</span>filter <span class="hl str">"annex"</span><span class="hl opt">]</span>
smudge <span class="hl opt">=</span> git-annex smudge <span class="hl kwb">--</span> <span class="hl opt">%</span>f
clean <span class="hl opt">=</span> git-annex smudge <span class="hl kwb">--clean --</span> <span class="hl opt">%</span>f
process <span class="hl opt">=</span> git-annex filter-process
<span class="hl opt">[</span>push<span class="hl opt">]</span>
followTags <span class="hl opt">=</span> true
<span class="hl opt">[</span>receive<span class="hl opt">]</span>
denyCurrentBranch <span class="hl opt">=</span> updateInstead
<span class="hl opt">[</span>remote <span class="hl str">"music-backup-one"</span><span class="hl opt">]</span>
url <span class="hl opt">= /</span>media<span class="hl opt">/</span>juanito<span class="hl opt">/</span>music-backup-one<span class="hl opt">/</span>music-library
fetch <span class="hl opt">= +</span>refs<span class="hl opt">/</span><span class="hl kwc">head</span>s<span class="hl opt">/*:</span>refs<span class="hl opt">/</span>remotes<span class="hl opt">/</span>music-backup-one<span class="hl opt">/*</span>
annex-uuid <span class="hl opt">=</span> <span class="hl num">0</span>b439b9f-16c7-4945-8cb2-9d6084677af3
<span class="hl opt">[</span>remote <span class="hl str">"music-backup-two"</span><span class="hl opt">]</span>
url <span class="hl opt">= /</span>media<span class="hl opt">/</span>juanito<span class="hl opt">/</span>music-backup-two<span class="hl opt">/</span>music-library
fetch <span class="hl opt">= +</span>refs<span class="hl opt">/</span><span class="hl kwc">head</span>s<span class="hl opt">/*:</span>refs<span class="hl opt">/</span>remotes<span class="hl opt">/</span>music-backup-two<span class="hl opt">/*</span>
annex-uuid <span class="hl opt">=</span> <span class="hl num">74</span>a6b33c-fea2-45ef-a0c5-b677b43bc85f
<span class="hl opt">[</span>remote <span class="hl str">"github"</span><span class="hl opt">]</span>
url <span class="hl opt">=</span> git@github.com<span class="hl opt">:</span>not-for-you.git
fetch <span class="hl opt">= +</span>refs<span class="hl opt">/</span><span class="hl kwc">head</span>s<span class="hl opt">/*:</span>refs<span class="hl opt">/</span>remotes<span class="hl opt">/</span>github<span class="hl opt">/*</span>
annex-ignore <span class="hl opt">=</span> true
<span class="hl opt">[</span>remote <span class="hl str">"itunes-media"</span><span class="hl opt">]</span>
annex-directory <span class="hl opt">= /</span>home<span class="hl opt">/</span>juanito<span class="hl opt">/</span>annex-remotes<span class="hl opt">/</span>itunes-media
annex-uuid <span class="hl opt">=</span> e18dbc8e-8e0f-47f2-8f33-e8eb9da97a62
skipFetchAll <span class="hl opt">=</span> true
annex-tracking-branch <span class="hl opt">=</span> main<span class="hl opt">:</span>Music
<span class="hl opt">[</span>remote <span class="hl str">"linux-media"</span><span class="hl opt">]</span>
annex-directory <span class="hl opt">= /</span>home<span class="hl opt">/</span>juanito<span class="hl opt">/</span>annex-remotes<span class="hl opt">/</span>linux-media
annex-uuid <span class="hl opt">=</span> b4efca25-5be6-4ea5-b0f9-207770abf21b
skipFetchAll <span class="hl opt">=</span> true
annex-tracking-branch <span class="hl opt">=</span> main<span class="hl opt">:</span>Music
Here is the contents of .git<span class="hl opt">/</span>config <span class="hl kwa">for</span> music-backup-one<span class="hl opt">:</span>
<span class="hl opt">[</span>core<span class="hl opt">]</span>
repositoryformatversion <span class="hl opt">=</span> <span class="hl num">0</span>
filemode <span class="hl opt">=</span> true
bare <span class="hl opt">=</span> false
logallrefupdates <span class="hl opt">=</span> true
<span class="hl opt">[</span>remote <span class="hl str">"desktop-music"</span><span class="hl opt">]</span>
url <span class="hl opt">= /</span>home<span class="hl opt">/</span>juanito<span class="hl opt">/</span>git<span class="hl opt">/</span>music-library
fetch <span class="hl opt">= +</span>refs<span class="hl opt">/</span><span class="hl kwc">head</span>s<span class="hl opt">/*:</span>refs<span class="hl opt">/</span>remotes<span class="hl opt">/</span>desktop-music<span class="hl opt">/*</span>
annex-uuid <span class="hl opt">=</span> <span class="hl num">0045</span>d866-c80e-43c1-9b1b-b15a8c97c1aa
<span class="hl opt">[</span>branch <span class="hl str">"adjusted/main(unlocked)"</span><span class="hl opt">]</span>
remote <span class="hl opt">=</span> desktop-music
merge <span class="hl opt">=</span> refs<span class="hl opt">/</span><span class="hl kwc">head</span>s<span class="hl opt">/</span>adjusted<span class="hl opt">/</span>main<span class="hl opt">(</span>unlocked<span class="hl opt">)</span>
<span class="hl opt">[</span>annex<span class="hl opt">]</span>
uuid <span class="hl opt">=</span> <span class="hl num">0</span>b439b9f-16c7-4945-8cb2-9d6084677af3
version <span class="hl opt">=</span> <span class="hl num">10</span>
adjustedbranchrefresh <span class="hl opt">=</span> true
<span class="hl opt">[</span>filter <span class="hl str">"annex"</span><span class="hl opt">]</span>
smudge <span class="hl opt">=</span> git-annex smudge <span class="hl kwb">--</span> <span class="hl opt">%</span>f
clean <span class="hl opt">=</span> git-annex smudge <span class="hl kwb">--clean --</span> <span class="hl opt">%</span>f
process <span class="hl opt">=</span> git-annex filter-process
<span class="hl opt">[</span>push<span class="hl opt">]</span>
followTags <span class="hl opt">=</span> true
<span class="hl opt">[</span>receive<span class="hl opt">]</span>
denyCurrentBranch <span class="hl opt">=</span> updateInstead
Here is the contents of .git<span class="hl opt">/</span>config <span class="hl kwa">for</span> music-backup-two<span class="hl opt">:</span>
<span class="hl opt">[</span>core<span class="hl opt">]</span>
repositoryformatversion <span class="hl opt">=</span> <span class="hl num">0</span>
filemode <span class="hl opt">=</span> true
bare <span class="hl opt">=</span> false
logallrefupdates <span class="hl opt">=</span> true
<span class="hl opt">[</span>remote <span class="hl str">"desktop-music"</span><span class="hl opt">]</span>
url <span class="hl opt">= /</span>home<span class="hl opt">/</span>juanito<span class="hl opt">/</span>git<span class="hl opt">/</span>music-library
fetch <span class="hl opt">= +</span>refs<span class="hl opt">/</span><span class="hl kwc">head</span>s<span class="hl opt">/*:</span>refs<span class="hl opt">/</span>remotes<span class="hl opt">/</span>desktop-music<span class="hl opt">/*</span>
annex-uuid <span class="hl opt">=</span> <span class="hl num">0045</span>d866-c80e-43c1-9b1b-b15a8c97c1aa
<span class="hl opt">[</span>branch <span class="hl str">"adjusted/main(unlocked)"</span><span class="hl opt">]</span>
remote <span class="hl opt">=</span> desktop-music
merge <span class="hl opt">=</span> refs<span class="hl opt">/</span><span class="hl kwc">head</span>s<span class="hl opt">/</span>adjusted<span class="hl opt">/</span>main<span class="hl opt">(</span>unlocked<span class="hl opt">)</span>
<span class="hl opt">[</span>annex<span class="hl opt">]</span>
uuid <span class="hl opt">=</span> <span class="hl num">74</span>a6b33c-fea2-45ef-a0c5-b677b43bc85f
version <span class="hl opt">=</span> <span class="hl num">10</span>
adjustedbranchrefresh <span class="hl opt">=</span> true
<span class="hl opt">[</span>filter <span class="hl str">"annex"</span><span class="hl opt">]</span>
smudge <span class="hl opt">=</span> git-annex smudge <span class="hl kwb">--</span> <span class="hl opt">%</span>f
clean <span class="hl opt">=</span> git-annex smudge <span class="hl kwb">--clean --</span> <span class="hl opt">%</span>f
process <span class="hl opt">=</span> git-annex filter-process
<span class="hl opt">[</span>push<span class="hl opt">]</span>
followTags <span class="hl opt">=</span> true
<span class="hl opt">[</span>receive<span class="hl opt">]</span>
denyCurrentBranch <span class="hl opt">=</span> updateInstead
<span class="hl slc"># End of transcript or log.</span>
</pre></div>
<h3>Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)</h3>
too aggressive in claiming "Transfer stalled"?http://git-annex.branchable.com/bugs/too_aggressive_in_claiming___34__Transfer_stalled__34____63__/yoh2024-01-04T20:23:32Z2024-01-04T20:23:32Z
<h3>Please describe the problem.</h3>
<p>Doing backup of data using <code>git annex move</code> from local to dropbox via <code>git-annex-remote-rclone</code>.</p>
<p>Full invocation here is</p>
<pre><code class="shell">( source ~/git-annexes/10.20231227+git24-gd37dbd62b8.env; OUR_LOG_FILE=~/.cache/rclone/logs/`date --iso-8601=seconds`-movedebug; RCLONE_LOG_LEVEL=DEBUG RCLONE_LOG_FILE=${OUR_LOG_FILE}_rclone.log git annex --debug move --to dandi-dandisets-dropbox sub-484677/sub-484677_ses-20210419T161140_behavior+ecephys+ogen.nwb 2>&1 | tee ${OUR_LOG_FILE}_git-annex.log ; )
</code></pre>
<p>under <code>/mnt/backup/dandi/dandisets/000363</code> . On git-annex side we see</p>
<pre><code class="shell">dandi@drogon:~/.cache/rclone/logs$ grep -4 'Transfer stalled' 2024-01-04T14:28:29-05:00-movedebug_git-annex.log
[2024-01-04 14:43:24.5722525] (Annex.TransferrerPool) < op 295000000000
75% 274.74 GiB 7 MiB/s 3h46m2024/01/04 14:44:49 DEBUG : Setting --log-level "DEBUG" from environment variable RCLONE_LOG_LEVEL="DEBUG"
2024/01/04 14:44:49 DEBUG : Setting --log-file "/home/dandi/.cache/rclone/logs/2024-01-04T14:28:29-05:00-movedebug_rclone.log" from environment variable RCLONE_LOG_FILE="/home/dandi/.cache/rclone/logs/2024-01-04T14:28:29-05:00-movedebug_rclone.log"
Transfer stalled
[2024-01-04 14:45:40.042053663] (Utility.Process) process [2329029] chat: /home/dandi/git-annexes/10.20231227+git24-gd37dbd62b8/usr/lib/git-annex.linux/git-annex ["transferrer","-c","annex.debug=true"]
</code></pre>
<p>and rclone logs for around that time point(s):</p>
<pre><code>2024/01/04 14:44:49 DEBUG : rclone: Version "v1.61.1-DEV" starting with parameters ["rclone" "copy" ".git/annex/tmp/SHA256E-s393314879887-S1000000000-C296--38c8c93c66ece7cbe8eeca9621ab72bf3968873e842f3e376ddda012eb0a0e5f.nwb" "dandi-dandisets-dropbox:dandi-dandisets/annexstore/b4f/757/"]
2024/01/04 14:44:49 DEBUG : Creating backend with remote ".git/annex/tmp/SHA256E-s393314879887-S1000000000-C296--38c8c93c66ece7cbe8eeca9621ab72bf3968873e842f3e376ddda012eb0a0e5f.nwb"
2024/01/04 14:44:49 DEBUG : Using config file from "/home/dandi/.rclone.conf"
2024/01/04 14:44:49 DEBUG : fs cache: adding new entry for parent of ".git/annex/tmp/SHA256E-s393314879887-S1000000000-C296--38c8c93c66ece7cbe8eeca9621ab72bf3968873e842f3e376ddda012eb0a0e5f.nwb", "/mnt/backup/dandi/dandisets/000363/.git/annex/tmp"
2024/01/04 14:44:49 DEBUG : Creating backend with remote "dandi-dandisets-dropbox:dandi-dandisets/annexstore/b4f/757/"
2024/01/04 14:44:49 DEBUG : fs cache: renaming cache item "dandi-dandisets-dropbox:dandi-dandisets/annexstore/b4f/757/" to be canonical "dandi-dandisets-dropbox:dandi-dandisets/annexstore/b4f/757"
2024/01/04 14:44:49 DEBUG : SHA256E-s393314879887-S1000000000-C296--38c8c93c66ece7cbe8eeca9621ab72bf3968873e842f3e376ddda012eb0a0e5f.nwb: Need to transfer - File not found at Destination
2024/01/04 14:44:50 DEBUG : SHA256E-s393314879887-S1000000000-C296--38c8c93c66ece7cbe8eeca9621ab72bf3968873e842f3e376ddda012eb0a0e5f.nwb: Uploading chunk 1/20
...
2024/01/04 14:45:36 DEBUG : SHA256E-s393314879887-S1000000000-C296--38c8c93c66ece7cbe8eeca9621ab72bf3968873e842f3e376ddda012eb0a0e5f.nwb: Uploading chunk 19/20
2024/01/04 14:45:38 DEBUG : SHA256E-s393314879887-S1000000000-C296--38c8c93c66ece7cbe8eeca9621ab72bf3968873e842f3e376ddda012eb0a0e5f.nwb: Uploading chunk 20/20
2024/01/04 14:45:39 INFO : Signal received: interrupt
2024/01/04 14:45:39 INFO : Dropbox root 'dandi-dandisets/annexstore/b4f/757': Committing uploads - please wait...
2024/01/04 14:45:39 INFO : Exiting...
</code></pre>
<p><details>
<summary>using older git-annex I got similar crash but on earlier chunk, so it is not pertinent to last chunk</summary></p>
<pre><code class="shell">2024/01/04 13:56:29 DEBUG : rclone: Version "v1.61.1-DEV" starting with parameters ["rclone" "copy" ".git/annex/tmp/SHA256E-s393314879887-S1000000000-C287--38c8c93c66ece7cbe8eeca9621ab72bf3968873e842f3e376ddda012eb0a0e5f.nwb" "dandi-dandisets-dropbox:dandi-dandisets/annexstore/b4f/757/"]
2024/01/04 13:56:29 DEBUG : Creating backend with remote ".git/annex/tmp/SHA256E-s393314879887-S1000000000-C287--38c8c93c66ece7cbe8eeca9621ab72bf3968873e842f3e376ddda012eb0a0e5f.nwb"
2024/01/04 13:56:29 DEBUG : Using config file from "/home/dandi/.rclone.conf"
2024/01/04 13:56:29 DEBUG : fs cache: adding new entry for parent of ".git/annex/tmp/SHA256E-s393314879887-S1000000000-C287--38c8c93c66ece7cbe8eeca9621ab72bf3968873e842f3e376ddda012eb0a0e5f.nwb", "/mnt/backup/dandi/dandisets/000363/.git/annex/tmp"
2024/01/04 13:56:29 DEBUG : Creating backend with remote "dandi-dandisets-dropbox:dandi-dandisets/annexstore/b4f/757/"
2024/01/04 13:56:30 DEBUG : fs cache: renaming cache item "dandi-dandisets-dropbox:dandi-dandisets/annexstore/b4f/757/" to be canonical "dandi-dandisets-dropbox:dandi-dandisets/annexstore/b4f/757"
2024/01/04 13:56:30 DEBUG : SHA256E-s393314879887-S1000000000-C287--38c8c93c66ece7cbe8eeca9621ab72bf3968873e842f3e376ddda012eb0a0e5f.nwb: Need to transfer - File not found at Destination
2024/01/04 13:56:30 DEBUG : SHA256E-s393314879887-S1000000000-C287--38c8c93c66ece7cbe8eeca9621ab72bf3968873e842f3e376ddda012eb0a0e5f.nwb: Uploading chunk 1/20
...
2024/01/04 13:57:08 DEBUG : SHA256E-s393314879887-S1000000000-C287--38c8c93c66ece7cbe8eeca9621ab72bf3968873e842f3e376ddda012eb0a0e5f.nwb: Uploading chunk 17/20
2024/01/04 13:57:10 DEBUG : SHA256E-s393314879887-S1000000000-C287--38c8c93c66ece7cbe8eeca9621ab72bf3968873e842f3e376ddda012eb0a0e5f.nwb: Uploading chunk 18/20
2024/01/04 13:57:11 DEBUG : SHA256E-s393314879887-S1000000000-C287--38c8c93c66ece7cbe8eeca9621ab72bf3968873e842f3e376ddda012eb0a0e5f.nwb: Uploading chunk 19/20
2024/01/04 13:57:14 INFO : Signal received: interrupt
2024/01/04 13:57:14 INFO : Dropbox root 'dandi-dandisets/annexstore/b4f/757': Committing uploads - please wait...
2024/01/04 13:57:14 INFO : Exiting...
</code></pre>
<p></details></p>
<p>So it seems that stall detection (I didn't look how it even decides on that) is flawed in case of large (chunked) files being transferred through external custom remotes which have no way to report progress as it is the case here in https://github.com/git-annex-remote-rclone/git-annex-remote-rclone/blob/master/git-annex-remote-rclone (filed <a href="https://github.com/git-annex-remote-rclone/git-annex-remote-rclone/issues/76">a bug report</a> to see that addressed).</p>
<h3>What version of git-annex are you using? On what operating system?</h3>
<p>first was a year old version, then tried with bleeding edge 10.20231227+git24-gd37dbd62b8</p>
No way to merge conflicts while on adjusted branchhttp://git-annex.branchable.com/bugs/No_way_to_merge_conflicts_while_on_adjusted_branch/2023-12-30T10:51:29Z2023-12-30T10:49:31Z
<h3>Please describe the problem.</h3>
<p>While on an adjusted branch, merge conflicts can neither be resolved manually nor with a git merge driver. See the transcript below.</p>
<h3>What version of git-annex are you using? On what operating system?</h3>
<p>git-annex version: 10.20230802
debian bullseye</p>
<h3>Please provide any additional information below.</h3>
<div class="highlight-sh"><pre class="hl">$ <span class="hl kwc">cat</span> ~<span class="hl opt">/</span>.gitconfig
...
<span class="hl opt">[</span>merge <span class="hl str">"commit"</span><span class="hl opt">]</span>
name <span class="hl opt">=</span> commit conflict markers
driver <span class="hl opt">=</span> <span class="hl str">"bash -c 'git merge-file %A %O %B; exit 0'"</span>
...
$ <span class="hl kwc">cat</span> .gitattributes
<span class="hl opt">*</span> annex.backend<span class="hl opt">=</span>SHA256E
study.org<span class="hl opt">*</span> merge<span class="hl opt">=</span>commit
dotfiles<span class="hl opt">/**</span> merge<span class="hl opt">=</span>commit
$ git annex sync
git-annex sync will change default behavior to operate on <span class="hl kwb">--content</span> <span class="hl kwa">in</span> a future version of git-annex. Recommend you explicitly use <span class="hl kwb">--no-content</span> <span class="hl opt">(</span>or <span class="hl kwb">-g</span><span class="hl opt">)</span> to prepare <span class="hl kwa">for</span> that change. <span class="hl opt">(</span>Or you can configure annex.synccontent<span class="hl opt">)</span>
commit
On branch adjusted<span class="hl opt">/</span>master<span class="hl opt">(</span>unlocked<span class="hl opt">)</span>
nothing to commit<span class="hl opt">,</span> working tree clean
ok
pull uni-lfs
<span class="hl opt">(</span>Merging into master...<span class="hl opt">)</span>
Auto-merging study.org
CONFLICT <span class="hl opt">(</span>content<span class="hl opt">):</span> Merge conflict <span class="hl kwa">in</span> study.org
Automatic merge failed<span class="hl opt">;</span> fix conflicts and <span class="hl kwa">then</span> commit the result.
<span class="hl opt">(</span>Merging into master...<span class="hl opt">)</span>
Auto-merging study.org
CONFLICT <span class="hl opt">(</span>content<span class="hl opt">):</span> Merge conflict <span class="hl kwa">in</span> study.org
Automatic merge failed<span class="hl opt">;</span> fix conflicts and <span class="hl kwa">then</span> commit the result.
failed
push uni-lfs
To xxx
<span class="hl opt">! [</span>rejected<span class="hl opt">]</span> master <span class="hl opt">-></span> synced<span class="hl opt">/</span>master <span class="hl opt">(</span>non-fast-forward<span class="hl opt">)</span>
error<span class="hl opt">:</span> failed to push some refs to <span class="hl str">'xxx'</span>
hint<span class="hl opt">:</span> Updates were rejected because a pushed branch tip is behind its remote
hint<span class="hl opt">:</span> counterpart. Check out this branch and integrate the remote changes
hint<span class="hl opt">: (</span>e.g. <span class="hl str">'git pull ...'</span><span class="hl opt">)</span> before pushing again.
hint<span class="hl opt">:</span> See the <span class="hl str">'Note about fast-forwards'</span> <span class="hl kwa">in</span> <span class="hl str">'git push --help'</span> <span class="hl kwa">for</span> details.
To xxx
<span class="hl opt">! [</span>rejected<span class="hl opt">]</span> master <span class="hl opt">-></span> master <span class="hl opt">(</span>non-fast-forward<span class="hl opt">)</span>
error<span class="hl opt">:</span> failed to push some refs to <span class="hl str">'xxx'</span>
hint<span class="hl opt">:</span> Updates were rejected because a pushed branch tip is behind its remote
hint<span class="hl opt">:</span> counterpart. Check out this branch and integrate the remote changes
hint<span class="hl opt">: (</span>e.g. <span class="hl str">'git pull ...'</span><span class="hl opt">)</span> before pushing again.
hint<span class="hl opt">:</span> See the <span class="hl str">'Note about fast-forwards'</span> <span class="hl kwa">in</span> <span class="hl str">'git push --help'</span> <span class="hl kwa">for</span> details.
Pushing to uni-lfs failed.
failed
sync<span class="hl opt">:</span> <span class="hl num">2</span> failed
$ git annex version
git-annex version<span class="hl opt">:</span> <span class="hl num">10.20230802</span>
build flags<span class="hl opt">:</span> Assistant Webapp Pairing Inotify DBus DesktopNotify TorrentParser MagicMime Benchmark Feeds Testsuite S3 WebDAV
dependency versions<span class="hl opt">:</span> aws-0.22 bloomfilter-2.0<span class="hl num">.1.0</span> cryptonite-0.26 DAV-1.3<span class="hl num">.4</span> feed-1.3<span class="hl num">.0.1</span> ghc-8.8<span class="hl num">.4</span> http-client-0.6<span class="hl num">.4.1</span> persistent-sqlite-2.10<span class="hl num">.6.2</span> torrent-10000.1<span class="hl num">.1</span> uuid-1.3<span class="hl num">.13</span> yesod-1.6<span class="hl num">.1.0</span>
key<span class="hl opt">/</span>value backends<span class="hl opt">:</span> SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL X<span class="hl opt">*</span>
remote types<span class="hl opt">:</span> git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs httpalso borg hook external
operating system<span class="hl opt">:</span> linux x86_64
supported repository versions<span class="hl opt">:</span> <span class="hl num">8 9 10</span>
upgrade supported from repository versions<span class="hl opt">:</span> <span class="hl num">0 1 2 3 4 5 6 7 8 9 10</span>
<span class="hl kwb">local</span> repository version<span class="hl opt">:</span> <span class="hl num">10</span>
<span class="hl slc"># End of transcript or log.</span>
</pre></div>