assistant can try to add too many files at once in batch add modegit-annexhttp://git-annex.branchable.com/bugs/Resource_exhausted/git-annexikiwiki2014-01-14T14:33:03Zcomment 1http://git-annex.branchable.com/bugs/Resource_exhausted/comment_1_a5ef7a62d4ed9365f9448520bb17e3b5/Laszlo2013-11-27T22:47:37Z2013-04-18T06:20:13Z
Once resource exhausted git-annex has git [defunct] process too.
comment 2http://git-annex.branchable.com/bugs/Resource_exhausted/comment_2_cdba2015e603f3c21f3e1697dd6fafcd/joey2013-11-27T22:47:37Z2013-04-23T19:44:04Z
<p>I have tried repeatedly to reproduce this problem, and I cannot.</p>
<p>git-annex does <em>not</em> keep every file open. It tends to have less than 10 open file descriptors at any one time.</p>
<p>I thought perhaps <code>lsof</code> opened every file, but it does not seem to, either.</p>
<p>So far, I have no indication that the problem had to do with git-annex at all. If some other program on the system opened a great many files, it could cause this to happen to git-annex.</p>
<p>You pasted a debug log that shows that the problem persisted for several minutes. So you should make it happen again, and in that time period, investigate what program has so many files open. You can do this with lsof, or, if lsof won't run, by looking in /proc/$pid/fd/</p>
<p>Or, of course, give me enough information to reproduce the problem. "I have 23459 files" isn't much help..</p>
I just reproduced thishttp://git-annex.branchable.com/bugs/Resource_exhausted/comment_3_747d16d050fdcf69dd3d2bc5ca469a2e/Adam2013-11-27T22:47:37Z2013-07-26T23:37:26Z
<p>On openSUSE 12.3 with this version (which I'm sure is horribly old):</p>
<pre><code>git-annex version: 4.20130314
local repository version: 3
default repository version: 3
supported repository versions: 3 4
upgrade supported from repository versions: 0 1 2
build flags: Assistant Webapp Pairing Testsuite S3 WebDAV Inotify DBus XMPP DNS
</code></pre>
<p>I ran 'git annex get' on a large repository, and got this:</p>
<pre><code>[snipped]
git-annex: /home/adam/music/.git/annex/transfer/failed/download/9c930baf-6b25-4294-b1f9-a90ba9663fb7/SHA1-s7227476--49d203960b062ec00c0d1d7042c4b6aa6720b976: openFile: resource exhausted (Too many open files)
failed
git-annex: runInteractiveProcess: pipe: Too many open files
git-annex: git: runInteractiveProcess: resource exhausted (Too many open files)
failed
git-annex: get: 1207 failed
</code></pre>
<p>Then I ran it again and saw that after every file retrieved, git-annex leaks another lockfile. lsof shows an ever increasing number of files like this:</p>
<pre><code>[snipped]
git-annex 32498 adam 86rR REG 253,3 0 787353 /home/adam/music/.git/annex/ssh/adam@atlantic.lock
git-annex 32498 adam 87rR REG 253,3 0 787353 /home/adam/music/.git/annex/ssh/adam@atlantic.lock
git-annex 32498 adam 88rR REG 253,3 0 787353 /home/adam/music/.git/annex/ssh/adam@atlantic.lock
git-annex 32498 adam 89rR REG 253,3 0 787353 /home/adam/music/.git/annex/ssh/adam@atlantic.lock
git-annex 32498 adam 90rR REG 253,3 0 787353 /home/adam/music/.git/annex/ssh/adam@atlantic.lock
git-annex 32498 adam 91rR REG 253,3 0 787353 /home/adam/music/.git/annex/ssh/adam@atlantic.lock
git-annex 32498 adam 92rR REG 253,3 0 787353 /home/adam/music/.git/annex/ssh/adam@atlantic.lock
git-annex 32498 adam 93r FIFO 0,8 0t0 28752703 pipe
git-annex 32498 adam 94r FIFO 0,8 0t0 28765910 pipe
</code></pre>
comment 4http://git-annex.branchable.com/bugs/Resource_exhausted/comment_4_1e9b74e60da57c3d5f08c1eb3801c1d2/joeyh.name2013-11-27T22:47:37Z2013-07-28T00:23:08Z
<p>Hmm, Adam your version is older than the bug reporter's version. OTOH, while there were several FD leak fixes after your version, none of them were to Annex.LockPool, which is what's used for the ssh lock files.</p>
<p>I can't reproduce it with <code>git annex get</code> and the current release.. can you?</p>
I just reproduced this with the latest versionhttp://git-annex.branchable.com/bugs/Resource_exhausted/comment_5_f55d933bce77fd2185ebd0cc46fe57ec/Michael2013-11-27T22:47:37Z2013-08-11T03:17:11Z
<p>At some point during a large copy, there's an ever increasing number of pipes in /proc/git-annex-pid/fd
As soon as it hits the limit (1023 in my case), copies start failing</p>
<pre><code>etc
r-x------ 1 michael michael 64 Aug 10 20:14 851 -> pipe:[2250609]
l-wx------ 1 michael michael 64 Aug 10 20:14 852 -> pipe:[2251549]
lr-x------ 1 michael michael 64 Aug 10 20:14 853 -> pipe:[2251550]
l-wx------ 1 michael michael 64 Aug 10 20:14 854 -> pipe:[2250612]
lr-x------ 1 michael michael 64 Aug 10 20:14 855 -> pipe:[2250613]
l-wx------ 1 michael michael 64 Aug 10 20:14 856 -> pipe:[2246639]
lr-x------ 1 michael michael 64 Aug 10 20:14 857 -> pipe:[2246640]
l-wx------ 1 michael michael 64 Aug 10 20:14 858 -> pipe:[2246642]
lr-x------ 1 michael michael 64 Aug 10 20:14 859 -> pipe:[2246643]
l-wx------ 1 michael michael 64 Aug 10 20:13 86 -> pipe:[2241378]
l-wx------ 1 michael michael 64 Aug 10 20:14 860 -> pipe:[2246645]
lr-x------ 1 michael michael 64 Aug 10 20:14 861 -> pipe:[2246646]
l-wx------ 1 michael michael 64 Aug 10 20:14 862 -> pipe:[2246648]
lr-x------ 1 michael michael 64 Aug 10 20:14 863 -> pipe:[2246649]
l-wx------ 1 michael michael 64 Aug 10 20:14 864 -> pipe:[2246653]
lr-x------ 1 michael michael 64 Aug 10 20:14 865 -> pipe:[2246654]
lr-x------ 1 michael michael 64 Aug 10 20:14 866 -> pipe:[2249407]
l-wx------ 1 michael michael 64 Aug 10 20:14 867 -> pipe:[2251789]
l-wx------ 1 michael michael 64 Aug 10 20:14 868 -> pipe:[2250627]
lr-x------ 1 michael michael 64 Aug 10 20:14 869 -> pipe:[2250628]
lr-x------ 1 michael michael 64 Aug 10 20:13 87 -> pipe:[2241379]
l-wx------ 1 michael michael 64 Aug 10 20:14 870 -> pipe:[2251778]
lr-x------ 1 michael michael 64 Aug 10 20:14 871 -> pipe:[2251779]
l-wx------ 1 michael michael 64 Aug 10 20:14 872 -> pipe:[2251781]
lr-x------ 1 michael michael 64 Aug 10 20:14 873 -> pipe:[2251782]
l-wx------ 1 michael michael 64 Aug 10 20:14 874 -> pipe:[2250635]
lr-x------ 1 michael michael 64 Aug 10 20:14 875 -> pipe:[2250636]
l-wx------ 1 michael michael 64 Aug 10 20:14 876 -> pipe:[2251575]
lr-x------ 1 michael michael 64 Aug 10 20:14 877 -> pipe:[2251576]
l-wx------ 1 michael michael 64 Aug 10 20:14 878 -> pipe:[2251785]
lr-x------ 1 michael michael 64 Aug 10 20:14 879 -> pipe:[2251786]
l-wx------ 1 michael michael 64 Aug 10 20:13 88 -> pipe:[2247853]
l-wx------ 1 michael michael 64 Aug 10 20:14 880 -> pipe:[2249430]
lr-x------ 1 michael michael 64 Aug 10 20:14 881 -> pipe:[2249431]
l-wx------ 1 michael michael 64 Aug 10 20:14 882 -> pipe:[2251581]
lr-x------ 1 michael michael 64 Aug 10 20:14 883 -> pipe:[2251582]
l-wx------ 1 michael michael 64 Aug 10 20:14 884 -> pipe:[2250653]
lr-x------ 1 michael michael 64 Aug 10 20:14 885 -> pipe:[2250654]
lr-x------ 1 michael michael 64 Aug 10 20:14 886 -> pipe:[2251790]
l-wx------ 1 michael michael 64 Aug 10 20:14 887 -> pipe:[2250670]
l-wx------ 1 michael michael 64 Aug 10 20:14 888 -> pipe:[2250663]
lr-x------ 1 michael michael 64 Aug 10 20:14 889 -> pipe:[2250664]
lr-x------ 1 michael michael 64 Aug 10 20:13 89 -> pipe:[2247854]
l-wx------ 1 michael michael 64 Aug 10 20:14 890 -> pipe:[2250668]
lr-x------ 1 michael michael 64 Aug 10 20:14 891 -> pipe:[2250669]
lr-x------ 1 michael michael 64 Aug 10 20:14 892 -> pipe:[2250671]
l-wx------ 1 michael michael 64 Aug 10 20:14 894 -> pipe:[2251601]
lr-x------ 1 michael michael 64 Aug 10 20:14 895 -> pipe:[2251602]
lr-x------ 1 michael michael 64 Aug 10 20:13 9 -> pipe:[2241268]
lr-x------ 1 michael michael 64 Aug 10 20:13 90 -> pipe:[2242273]
l-wx------ 1 michael michael 64 Aug 10 20:13 91 -> pipe:[2245776]
etc
</code></pre>
comment 6http://git-annex.branchable.com/bugs/Resource_exhausted/comment_6_26c98fca45b029a527f9684873db4be5/Michael2013-11-27T22:47:37Z2013-08-11T03:20:37Z
<p>with every</p>
<pre><code>git annex
copy blah/blah/blah (to testremote...)
git-annex: runInteractiveProcess: pipe: Too many open files
failed
</code></pre>
<p>The number of open fd's by git-annex increases by 1.</p>
<p>4.20130802 built with cabal on Ubuntu 13.04</p>
comment 7http://git-annex.branchable.com/bugs/Resource_exhausted/comment_7_8bab413b472f900e04977db2bc3951b6/Michael2013-11-27T22:47:37Z2013-08-11T03:21:45Z
...increases in the failing case, that is. This doesn't happen all the time.
comment 8http://git-annex.branchable.com/bugs/Resource_exhausted/comment_8_e9bec0b80179b1229b6af0979a21c727/Michael2013-11-27T22:47:37Z2013-08-11T20:44:19Z
<p>It also looks like the location log has got corrupted (files are actually present, but not recorded in the location log) somewhere along the lines as I was trying to get/drop to figure out what's going on.
Explicitly dropping files then getting files fixes the location log issue.</p>
comment 9http://git-annex.branchable.com/bugs/Resource_exhausted/comment_9_419e24e0b91f569294ece28c42daa246/joeyh.name2013-11-27T22:47:37Z2013-08-24T19:10:21Z
<p>@Michael how large a copy are you doing? And what kind of remote are you copying the files to?
It would be helpful if you could be more specific about something I could do to reproduce the problem. Without a test case, I am unlikely to fix the bug. With a test case, I'd be surprised if it took long to fix it.</p>
<p>If you have a process running that is experiencing the problem, you can also narrow it down a <em>lot</em> by looking at what these leaking pipe file descriptors are pipes to. For example, if you have:</p>
<p>lr-x------ 1 michael michael 64 Aug 10 20:14 895 -> pipe:[2251602]</p>
<p>You can run <code>find /proc/ -ls 2251602</code> and find the process at other end of the pipe, and look its pid up in ps to see what command it is.</p>
comment 10http://git-annex.branchable.com/bugs/Resource_exhausted/comment_10_bccf9528ffe963154c92ce49762e7ea6/Michael2013-11-27T22:47:37Z2013-08-26T18:33:40Z
<p>@Joey: it was a "pretty large" transfer, several hundred gigabytes in perhaps ~100000 files. The copying was going to a GPG-encrypted directory remote.
The error only happened once or twice so far. Point taken about find in /proc; I'll do that if it happens next time.</p>
comment 11http://git-annex.branchable.com/bugs/Resource_exhausted/comment_11_24ad3a76a25a787ac59e2c5270709e72/joeyh.name2013-12-16T17:20:35Z2013-12-16T17:20:34Z
<p>I now understand the problem described in comment 6, where once it started failing, it would leak one file descriptor per failure.</p>
<p>I think that failure mode was fixed by accident in the changes in <a href="http://source.git-annex.branchable.com/?p=source.git;a=commitdiff;h=2fd63f3cfac705f0a18f4bcbe0489ce8ea1800d7">2fd63f3cfac705f0a18f4bcbe0489ce8ea1800d7</a>.</p>
<p>This doesn't explain what would open so many files to get it into that failure mode, however.</p>
Having "too many open files" message show uphttp://git-annex.branchable.com/bugs/Resource_exhausted/comment_12_1943ff77f3bccf885229ecc10c82399d/Tim2014-01-13T14:57:13Z2014-01-13T14:57:11Z
<p>This is happening to me now about twice per day. I've got a repo that is completely synced already and I'm working on a project in IntelliJ. When I check git-annex I often see this message when it is doing a startup scan or consistency check.</p>
<p>The message can be seen here: http://imgur.com/Xb4LA73</p>
<p>Are there logs somewhere that I can gather and supply to you to help track this down? I'm using the 2014-01-07 release of git-annex for Mac OS.</p>
Happened four times yesterdayhttp://git-annex.branchable.com/bugs/Resource_exhausted/comment_13_c3cac0717232e04e89df62efa1db0870/Tim2014-01-14T14:33:03Z2014-01-14T14:33:02Z
Maybe you can e-mail me and we can discuss this?