I was dumping ~gigs of files of approximately 3-6megs a pop (my music collection) so I could track the files that I want to listen to when I'm on the go. I had the git watch command running from the assistant branch.
I was getting something along the lines of...
/Users/jtang/annex/.git/annex/tmp/: openTempFile: resource exhausted (Too many open files)
and
git-annex: createPipe: resource exhausted (Too many open files)
I also noticed that I somehow ended up with 256 ssh-agent's running on one of my machines, I'm not sure if the two issues are related or not, I had not noticed this type of behaviour up until recently.
Also this was appearing in the logs
x00:annex jtang$ tail -f .git/annex/daemon.log
(scanning...) Already up-to-date.
kqueue: Too many open files
To be precise, I suspect that the kqueue limit is 256, I had 325 files in the 'queue', I ended up doing a git annex add manually and all was fine.
This affects BSD systems that use Kqueue. It no longer affects OSX, since we use FSEvents there instead. --Joey
Yes, this is a known problem with kqueue, it has to keep every directory in the tree open. On inotify I have a note that it may need to fork off extra watcher processes to deal with this. Of course that adds significant complication.
In the meantime, you may be able to increase your system's maximum allowed number of open files per process somehow.
(I doubt that the ssh-agent is related; git-annex does not use ssh-agent directly anyway..)
Doing,
Somewhat works for me, git-annex watch at least starts up and takes a while to scan the directory, but it's not ideal. Also, creating files seems to work okay, when I remove a file the changes don't seem to get pushed across my other repos, running a sync on the remote repo fixes things.
Jimmy, sounds like I could use something like this to get the current limit:
Probably prints "sysctl kern.maxfilesperproc = 256" or such.. can you verify? Once I have the limit, I can make the kqueue code use subset of it, and print out a message when it needs to be increased, like the inotify code does.
(Also, the kqueue code only opens directories, not files, so unless you have 400000 directories, that's a little high.)
On file removal not propigating, does this still happen? When you remove a file does a git commit automatically happen, or is that broken with kqueue?
In relation to the system limits,
Also, the maxfiles for the whole system is
the above was the defaults as far as I recall. What you probably would be interested is the ulimits that the user see
I would imagine the limit that you are looking for is 256. Hope this helps.
On the point about deletions not being propagated, it does do a commit. I suspect that the kqueue code is just not picking up the changes and pushing the changes out. The watch command on a single annex with no remotes functions as expected.
On kFreeBSD, I get this:
But ulimit still has 1024 limit, so you'd need to adjust both, as root. Messy..
& it's killing my assistant.
Found a workaround here: https://github.com/mockko/livereload/issues/1
This fixes it for now:
ulimit -n 4096
Just posting here in case anybody else runs into this issue and, like me, finds themselves here via search engine.