bugs/Race condition between watch/assistant and addurl
git-annex
http://git-annex.branchable.com/bugs/Race_condition_between_watch__47__assistant_and_addurl/
git-annex
ikiwiki
2014-04-02T20:17:19Z
Another race condition
http://git-annex.branchable.com/bugs/Race_condition_between_watch__47__assistant_and_addurl/comment_1_8f56b8661a600729d7a9d569e8a0ba70/
Johan
2014-03-31T03:42:42Z
2014-03-31T03:42:42Z
<p>Here’s another race condition which seems related:</p>
<div class="highlight-sh"><pre class="hl"><span class="hl opt">%</span> git annex addurl <span class="hl str">'quvi:http://youtu.be/-CbFj9K9AQg'</span>
addurl David_Raymond_Christiansen___Dependently_Typed_Programming_in_Idris___A_Demo.webm
<span class="hl kwb">--2014-03-31</span> <span class="hl num">05</span><span class="hl opt">:</span><span class="hl num">45</span><span class="hl opt">:</span><span class="hl num">49</span><span class="hl kwb">--</span> http<span class="hl opt">://</span>r4---sn-oxc0a5-ixae.googlevideo.com<span class="hl opt">/</span>videoplayback?<span class="hl opt"><</span>cut<span class="hl opt">></span>
Resolving r4---sn-oxc0a5-ixae.googlevideo.com <span class="hl opt">(</span>r4---sn-oxc0a5-ixae.googlevideo.com<span class="hl opt">)</span>... <span class="hl num">83.145.196.143</span><span class="hl opt">,</span> <span class="hl num">2001</span><span class="hl opt">:</span><span class="hl num">1</span>bc8<span class="hl opt">:</span><span class="hl num">100</span><span class="hl opt">:</span><span class="hl num">1</span>b<span class="hl opt">::</span>f
Connecting to r4---sn-oxc0a5-ixae.googlevideo.com <span class="hl opt">(</span>r4---sn-oxc0a5-ixae.googlevideo.com<span class="hl opt">)</span>|<span class="hl num">83.145.196.143</span>|<span class="hl opt">:</span><span class="hl num">80</span>... connected.
HTTP request sent<span class="hl opt">,</span> awaiting response... <span class="hl num">302</span> Found
Location<span class="hl opt">:</span> http<span class="hl opt">://</span>r13---sn-5go7dn7s.googlevideo.com<span class="hl opt">/</span>videoplayback?<span class="hl opt"><</span>cut<span class="hl opt">> [</span>following<span class="hl opt">]</span>
<span class="hl kwb">--2014-03-31</span> <span class="hl num">05</span><span class="hl opt">:</span><span class="hl num">45</span><span class="hl opt">:</span><span class="hl num">49</span><span class="hl kwb">--</span> http<span class="hl opt">://</span>r13---sn-5go7dn7s.googlevideo.com<span class="hl opt">/</span>videoplayback?<span class="hl opt"><</span>cut<span class="hl opt">></span>
Resolving r13---sn-5go7dn7s.googlevideo.com <span class="hl opt">(</span>r13---sn-5go7dn7s.googlevideo.com<span class="hl opt">)</span>... <span class="hl num">173.194.48.18</span><span class="hl opt">,</span> <span class="hl num">2</span>a00<span class="hl opt">:</span><span class="hl num">1450</span><span class="hl opt">:</span><span class="hl num">400</span>f<span class="hl opt">::</span><span class="hl num">12</span>
Connecting to r13---sn-5go7dn7s.googlevideo.com <span class="hl opt">(</span>r13---sn-5go7dn7s.googlevideo.com<span class="hl opt">)</span>|<span class="hl num">173.194.48.18</span>|<span class="hl opt">:</span><span class="hl num">80</span>... connected.
HTTP request sent<span class="hl opt">,</span> awaiting response... <span class="hl num">200</span> OK
Length<span class="hl opt">:</span> <span class="hl num">320557466</span> <span class="hl opt">(</span><span class="hl num">306</span>M<span class="hl opt">) [</span>video<span class="hl opt">/</span>webm<span class="hl opt">]</span>
Saving to<span class="hl opt">:</span> ‘<span class="hl opt">/</span>home<span class="hl opt">/</span>ion<span class="hl opt">/</span>nobackup<span class="hl opt">/</span>media<span class="hl opt">/</span>video<span class="hl opt">/</span>.git<span class="hl opt">/</span>annex<span class="hl opt">/</span>tmp<span class="hl opt">/</span>URL--quvi<span class="hl opt">&</span>chttp<span class="hl opt">&</span>c<span class="hl opt">%%</span>youtu.be<span class="hl opt">%</span><span class="hl kwb">-CbFj9K9AQg</span>’
<span class="hl num">100</span><span class="hl opt">%[======================================>]</span> <span class="hl num">320</span><span class="hl opt">,</span><span class="hl num">557</span><span class="hl opt">,</span><span class="hl num">466 895</span>KB<span class="hl opt">/</span>s <span class="hl kwa">in</span> <span class="hl num">6</span>m <span class="hl num">14</span>s
<span class="hl num">2014</span><span class="hl kwb">-03-31</span> <span class="hl num">05</span><span class="hl opt">:</span><span class="hl num">52</span><span class="hl opt">:</span><span class="hl num">04</span> <span class="hl opt">(</span><span class="hl num">837</span> KB<span class="hl opt">/</span>s<span class="hl opt">) -</span> ‘<span class="hl opt">/</span>home<span class="hl opt">/</span>ion<span class="hl opt">/</span>nobackup<span class="hl opt">/</span>media<span class="hl opt">/</span>video<span class="hl opt">/</span>.git<span class="hl opt">/</span>annex<span class="hl opt">/</span>tmp<span class="hl opt">/</span>URL--quvi<span class="hl opt">&</span>chttp<span class="hl opt">&</span>c<span class="hl opt">%%</span>youtu.be<span class="hl opt">%</span><span class="hl kwb">-CbFj9K9AQg</span>’ saved <span class="hl opt">[</span><span class="hl num">320557466</span><span class="hl opt">/</span><span class="hl num">320557466</span><span class="hl opt">]</span>
<span class="hl opt">(</span>Recording state <span class="hl kwa">in</span> git...<span class="hl opt">)</span>
fatal<span class="hl opt">:</span> Unable to create <span class="hl str">'/home/ion/nobackup/media/video/.git/index.lock'</span><span class="hl opt">:</span> File exists.
If no other git process is currently running<span class="hl opt">,</span> this probably means a
git process crashed <span class="hl kwa">in</span> this repository earlier. Make sure no other git
process is running and remove the <span class="hl kwc">file</span> manually to <span class="hl kwb">continue</span>.
git-annex<span class="hl opt">:</span> user error <span class="hl opt">(</span><span class="hl kwc">xargs</span> <span class="hl opt">[</span><span class="hl str">"-0"</span><span class="hl opt">,</span><span class="hl str">"git"</span><span class="hl opt">,</span><span class="hl str">"--git-dir=/home/ion/nobackup/media/video/.git"</span><span class="hl opt">,</span><span class="hl str">"--work-tree=/home/ion/nobackup/media/video"</span><span class="hl opt">,</span><span class="hl str">"-c"</span><span class="hl opt">,</span><span class="hl str">"core.bare=false"</span><span class="hl opt">,</span><span class="hl str">"add"</span><span class="hl opt">,</span><span class="hl str">"--"</span><span class="hl opt">]</span> exited <span class="hl num">123</span><span class="hl opt">)</span>
failed
git-annex<span class="hl opt">:</span> addurl<span class="hl opt">:</span> <span class="hl num">1</span> failed
<span class="hl opt">%</span> <span class="hl kwc">ls</span> <span class="hl kwb">-l</span> <span class="hl opt">/</span>home<span class="hl opt">/</span>ion<span class="hl opt">/</span>nobackup<span class="hl opt">/</span>media<span class="hl opt">/</span>video<span class="hl opt">/</span>.git<span class="hl opt">/</span>index.lock
<span class="hl kwc">ls</span><span class="hl opt">:</span> cannot access <span class="hl opt">/</span>home<span class="hl opt">/</span>ion<span class="hl opt">/</span>nobackup<span class="hl opt">/</span>media<span class="hl opt">/</span>video<span class="hl opt">/</span>.git<span class="hl opt">/</span>index.lock<span class="hl opt">:</span> No such <span class="hl kwc">file</span> or directory
</pre></div>
<p>The only lines in daemon.log from that time:</p>
<div class="highlight-sh"><pre class="hl"><span class="hl opt">[</span><span class="hl num">2014</span><span class="hl kwb">-03-31</span> <span class="hl num">05</span><span class="hl opt">:</span><span class="hl num">52</span><span class="hl opt">:</span><span class="hl num">04</span> EEST<span class="hl opt">]</span> Committer<span class="hl opt">:</span> Committing changes to git
<span class="hl opt">[</span><span class="hl num">2014</span><span class="hl kwb">-03-31</span> <span class="hl num">05</span><span class="hl opt">:</span><span class="hl num">52</span><span class="hl opt">:</span><span class="hl num">04</span> EEST<span class="hl opt">]</span> Pusher<span class="hl opt">:</span> Syncing with heh.<span class="hl kwa">fi</span>
Already up-to-date.
To heh.<span class="hl kwa">fi</span><span class="hl opt">:/</span>storage<span class="hl opt">/</span>ion<span class="hl opt">/</span>media<span class="hl opt">/</span>video
<span class="hl num">3</span>df241b.<span class="hl num">.095</span>d6c3 git-annex <span class="hl opt">-></span> synced<span class="hl opt">/</span>git-annex
<span class="hl num">10</span>b3166.<span class="hl num">.98074</span>c1 annex<span class="hl opt">/</span>direct<span class="hl opt">/</span>master <span class="hl opt">-></span> synced<span class="hl opt">/</span>master
Already up-to-date.
</pre></div>
comment 2
http://git-annex.branchable.com/bugs/Race_condition_between_watch__47__assistant_and_addurl/comment_2_46dc67bdcd174cd50ccc421ec56735ad/
joeyh.name
2014-04-02T20:17:19Z
2014-04-02T20:17:19Z
<p>These races look beniegn, as far as I can see it doesn't cause any data to be lost, or indeed anything to happen that wouldn't have happened if addurl had been run without the assistant running.</p>
<p>The first race probably has addurl and the assistant both trying to move the file object into the annex at the same time. One wins and moves it; the other loses and sulks.</p>
<p>The second race has addurl trying to <code>git add</code> the file, while the assistant has already noticed the file appeared, <code>git add</code>ed it, and committed the add.</p>
<p>The only way to really avoid these races would be to add a lot of lock checking. Or just make <code>git annex addurl</code> and presumably also <code>git annex add</code> and maybe several other commands refuse to run when the assistant is running.</p>