bugs/git annex assistant exits with failuregit-annexhttp://git-annex.branchable.com/bugs/git_annex_assistant_exits_with_failure/git-annexikiwiki2017-02-17T03:23:46Zcomment 1http://git-annex.branchable.com/bugs/git_annex_assistant_exits_with_failure/comment_1_247d4da208cd700851d48f13534d5f7c/joey2017-02-13T21:42:21Z2017-02-13T20:08:32Z
<p>"ExitFailure (-1073741819)" looks kind of like an overflow of the exit
status code, which is very strange.</p>
<p>Is there an error message output anywhere when this happens?</p>
<p>It looks like you might have a ton of files in this repository, and that
might have something to do with the memory use and load.</p>
comment 2http://git-annex.branchable.com/bugs/git_annex_assistant_exits_with_failure/comment_2_c6c04006b1d9aa6be92f5a970ba60618/lasitus2017-02-14T18:46:22Z2017-02-14T18:46:22Z
<p>""ExitFailure (-1073741819)" looks kind of like an overflow of the exit status code, which is very strange."
It isn't the only number I have received. I believe it was 5 on one run and 3 on another, but that is just from memory. I don't think the error was ever printed to the daemon.log. I can try and run it again and get more definite results.</p>
<p>"Is there an error message output anywhere when this happens?"
I gave you the end of daemon.log and everything that was printed after the crash. Any other logs you would like?</p>
<p>"It looks like you might have a ton of files in this repository, and that might have something to do with the memory use and load."
After restarting and getting it going again, it was happily moving along with not much memory usage and 0.5 load. It seemed like both the client and the server were hung up on something that time. I had left it to chew on the files for quite some time. (That time the client didn't crash, but it stopped sensing any changes. Restarts of the assistant also didn't fix the issue, only a reboot on the Windows side.) I suppose I could have found whatever was running instead in task manager, but I'm saying a ctrl-c and apparent restart didn't help.</p>
comment 3http://git-annex.branchable.com/bugs/git_annex_assistant_exits_with_failure/comment_3_eabad530a55adfa4be3eae0f743d3309/joey2017-02-14T18:56:21Z2017-02-14T18:50:02Z
<p>With a weird crash like this, I was thinking there might be an error message
output to the console. I guess you showed the full console output though.</p>
<p>Was the 85% of system memory happening on Windows or on Linux?</p>
<p>I have no clues how to debug this. If there's useful information anywhere,
it's probably in some windows crash log or system log thing that I know
nothing about.</p>
comment 4http://git-annex.branchable.com/bugs/git_annex_assistant_exits_with_failure/comment_4_204764bb40509d8650386227ea166a6e/lasitus2017-02-14T19:08:37Z2017-02-14T19:08:38Z
<p>The high memory use and load were on the Linux side.</p>
<p>Again these were two different behaviors on separate runs. Most runs, the linux side was fine and the windows side crashed. This one I can reproduce easily by just copying my data into an empty repo. I'll try it a few times later tonight and see if it always happens on a particular file.</p>
<p>One time, the windows side stopped sensing files and the Linux side had heavy cpu usage and high memory usage) Sadly, I don't recall the exact time as the hang up happened over night, so hard to find in the log file. So, I guess I will focus on the first one.</p>
comment 5http://git-annex.branchable.com/bugs/git_annex_assistant_exits_with_failure/comment_5_0453ae9123b359757e63c6eb5f12811b/lasitus2017-02-15T02:17:25Z2017-02-15T02:17:25Z
<p>Got a decent result this last run:</p>
<p>Console output</p>
<pre><code>[2017-02-14 20:40:25.3493472] logging to .git\annex\daemon.log
[2017-02-14 20:40:25.3493472] chat: C:\Program Files (x86)\Git\usr\bin\git-annex.exe ["assistant","--debug"]
[2017-02-14 20:47:21.2931135] process done ExitFailure 1
</code></pre>
<p>daemon.log ending</p>
<pre><code>[2017-02-14 20:47:17.4915947] process done ExitSuccess
[2017-02-14 20:47:19.7279831] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","core.bare=false","symbolic-ref","-q","HEAD"]
[2017-02-14 20:47:19.7436092] process done ExitSuccess
[2017-02-14 20:47:19.7436092] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","core.bare=false","show-ref","refs/heads/annex/direct/master"]
[2017-02-14 20:47:19.7748596] process done ExitSuccess
[2017-02-14 20:47:19.7748596] Pusher: pushing to [Remote { name ="origin" }]
[2017-02-14 20:47:19.8217363] Committer: queued Upload UUID "b77aaec0-e75b-4eec-a08c-b09cbe2947d5" LR Main\LR Main Smart Previews.lrdata\1\117C\117C0B4F-1D99-4DF6-9E7B-7C0B655876D9.dng Nothing : new file created
[2017-02-14 20:47:19.8529873] call: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","core.bare=false","push","origin","+git-annex:synced/git-annex","annex/direct/master:synced/master"]
[2017-02-14 20:47:19.9154902] Transferrer: Transferring: Upload UUID "b77aaec0-e75b-4eec-a08c-b09cbe2947d5" LR Main\LR Main Smart Previews.lrdata\1\117C\117C0B4F-1D99-4DF6-9E7B-7C0B655876D9.dng Nothing
[2017-02-14 20:47:19.9154902] Watcher: add LR Main\LR Main Smart Previews.lrdata\1\162B\162B90D0-CC7C-4BD9-AC2B-BA2D4C72B555.dng
117C0B4F-1D99-4DF6-9E7B-7C0B655876D9.dng
1,792,000 100% 152.52MB/s 0:00:00 (xfr#1, to-chk=0/1)
git-annex.exe: out of memory
(recording state in git...)
To storage.lan:/srv/annex/pictures
0623558..669c99c git-annex -> synced/git-annex
6309934..b8e043d annex/direct/master -> synced/master
</code></pre>
comment 6http://git-annex.branchable.com/bugs/git_annex_assistant_exits_with_failure/comment_6_5b0310038a8b45d2083f39ea75c1a979/lasitus2017-02-15T16:01:38Z2017-02-15T16:01:38Z
Note: The last run was the Windows side
comment 7http://git-annex.branchable.com/bugs/git_annex_assistant_exits_with_failure/comment_7_63b3ec113d52501237abbe6ee1ef5fa5/lasitus2017-02-17T03:23:46Z2017-02-17T03:23:46Z
<p>Ok, I have a script that generates the error. This generates a repository and 30 GB of random binary files with many folders 2 layers deep. Just put in an empty folder and run with python. No remotes are necessary. This was run in Windows 10 in a git bash window.</p>
<pre><code>#!/usr/bin/env python
import logging
import os
import shutil
import subprocess
import uuid
logging.basicConfig(level=logging.DEBUG)
repositoryPath = os.path.abspath("./bigRepoTest")
os.makedirs(repositoryPath)
subprocess.call("git init", cwd=repositoryPath)
subprocess.call("git annex init pc", cwd=repositoryPath)
def makeRandomDirectories(level1FolderCount, level2FolderCount, fileCount):
for directoryIndex in range(0, level1FolderCount):
logging.info("Adding top level folder " + str(directoryIndex + 1) + " of " + str(level1FolderCount))
newDirectory = os.path.join(repositoryPath, str(uuid.uuid1()))
os.makedirs(newDirectory)
for directoryIndex in range(0, level2FolderCount):
newNestedDirectory = os.path.join(newDirectory, str(uuid.uuid1()))
os.makedirs(newNestedDirectory)
for fileIndex in range(0, fileCount):
newFile = os.path.join(newNestedDirectory, str(uuid.uuid1()) + ".bin")
with open(newFile, 'wb') as fileOut:
fileOut.write(os.urandom(500000))
makeRandomDirectories(32, 1000, 1)
with open(os.path.join(repositoryPath, "assistant.log"), 'w') as output:
subprocess.Popen(["git", "annex", "assistant", "--debug"], cwd=repositoryPath, stdout=output, stderr=output)
makeRandomDirectories(32, 1000, 1)
subprocess.call("tail -f daemon.log", cwd=os.path.join(repositoryPath, ".git", "annex"))
</code></pre>