git annex assistant leaks a lot of memory

Hi,

I have a git-annex repository on a server managed by git annex assistant in indirect mode and as a backup group serving my org-mode files (organizer module for emacs) which are plain text files no more than a 100kb.

I also have 2 clients in direct mode to sync the files across my computers.

I save my files fairly often so the assistant might be a bit overwhelmed but I think it can managed that. At least it used to.

Since maybe a couple months, the assistant running on the server has been leaking nearly 1Gb each day and I suspect it rises whenever I push files to it.

RSS Atom

comment 1

Does the assistant's memory use go back to being small when you stop and restart the daemon, or does it immediately go back up?

Are there a lot of files in .git/annex/transfer? One possibility is it might be getting flooded with failed/pending file transfers.

If you feel up to building git-annex from source, you could enable the EKG remote monitoring interface. This would let you see graphs of the memory use, and probably get a good idea of what's using memory. Or, you could do a full memory profiling build of git-annex, which will let us nail down exactly what's using the memory.

See ?this page for instructions for building git-annex for EKG and memory profiling. If it's too hard for you to do that, I can probably provide you with a prebuilt version with memory profiling.

Comment by joey — Thu Nov 6 19:53:02 2014

Remove comment

comment 2

Hi,

Yes, the assistant memory goes back to normal when I do git annex assistant --stop && git annex assistant. But when I look at htop the next day, it has already taken 1Gb.

There are quite a lot of files in .git/annex/transfer… wc reports more than 200 entries, mainly in the upload folder, although the repository only manages 10 or so files.

I tried building git-annex using cabal but cabal configure -fEKG exits with the code 9 and even without -fEKG cabal refuses to install some of the dependencies. I'll try again some other time…

Comment by renaud [casenave.fr] — Sat Nov 8 07:18:48 2014

Remove comment

comment 3

Well it's been a week and still no luck compiling git-annex using cabal

I'm blocked here (at pretty much the start):

% cabal install --only-dependencies
Resolving dependencies...
cabal: Could not resolve dependencies:
trying: git-annex-5.20141125 (user goal)
trying: git-annex-5.20141125:+tahoe
trying: git-annex-5.20141125:+dns
trying: aeson-0.6.0.2/installed-5b6... (dependency of
git-annex-5.20141125:+tahoe)
trying: vector-0.9.1/installed-5fe... (dependency of
aeson-0.6.0.2/installed-5b6...)
trying: dns-1.4.4 (dependency of git-annex-5.20141125:+dns)
next goal: network (dependency of dns-1.4.4)
rejecting: network-2.3.0.13/installed-0e3... (conflict:
git-annex-5.20141125:network-uri => network(>=2.6))
trying: network-2.6.0.2
next goal: conduit-extra (dependency of dns-1.4.4)
rejecting: conduit-extra-1.1.4.2, 1.1.4.1, 1.1.4, 1.1.3.4, 1.1.3.3, 1.1.3.2,
1.1.3.1, 1.1.3, 1.1.2, 1.1.1, 1.1.0.4 (conflict: vector =>
primitive==0.4.1/installed-505..., conduit-extra => primitive>=0.5)
rejecting: conduit-extra-1.1.0.3, 1.1.0.2, 1.1.0.1 (conflict: vector =>
primitive==0.4.1/installed-505..., conduit-extra => primitive>=0.5 && <0.6)
rejecting: conduit-extra-1.1.0 (conflict: network==2.6.0.2, conduit-extra =>
network>=2.3 && <2.5)
rejecting: conduit-extra-1.0.0.1, 1.0.0, 0.1.7, 0.1.6, 0.1.5, 0.1.4, 0.1.3,
0.1.2, 0.1.1, 0.1.0 (conflict: dns => conduit-extra>=1.1)
zsh: exit 1     cabal install --only-dependencies

Should I just wait for the maintainers to fix the dependencies? Or is there another way?

Comment by renaud [casenave.fr] — Sat Nov 15 02:07:38 2014

Remove comment

comment 4

I suffer from the same issue.

In ~/.profile I put the line git-annex assistant --quiet --autostart --startdelay=5s > /dev/null &.

Htop shows:

26291 xxx      20   0 6584M 3123M  9864 S  0.0 81.6 29h29:17 git-annex assistant --startdelay=5s
26292 xxx      20   0 6584M 3123M  9864 S  0.0 81.6 12h38:32 git-annex assistant --startdelay=5s
26293 xxx      20   0 6584M 3123M  9864 S  0.0 81.6 21h45:49 git-annex assistant --startdelay=5s
26321 xxx      20   0 6584M 3123M  9864 D  0.0 81.6 18:34.12 git-annex assistant --startdelay=5s
21019 xxx      20   0 6584M 3123M  9864 S  0.0 81.6  0:00.08 git-annex assistant --startdelay=5s
21908 xxx      20   0 6584M 3123M  9864 S  0.0 81.6  0:00.00 git-annex assistant --startdelay=5s
26290 xxx      20   0 6584M 3123M  9864 S  0.0 81.6 64h33:50 git-annex assistant --startdelay=5s

Comment by Dennis — Thu Dec 11 10:58:21 2014

Remove comment

Add a comment