Hi,
I have run the git annex repair command on one of my repositories and it has been running for 38h hours now. I have monitored the process and its sons with htop and it seems that git annex repair is stuck.
It repeatedly launches the following command:
git --git-dir=/home/vincent/photos2/.git --work-tree=/home/vincent/photos2 ls-tree --full-tree -z -r -- 5fe5193e079631c0ceac0688ae0a6c1636491b61
I have tried to execute it by hand and it produces a lot of output. I have redirected it into some file in order to count the content produced by this command.
vincent@berlioz:~/photos2/.git$ git --git-dir=/home/vincent/photos2/.git --work-tree=/home/vincent/photos2 ls-tree --full-tree -z -r -- 5fe5193e079631c0ceac0688ae0a6c1636491b61 > /tmp/ls-tree
vincent@berlioz:/tmp$ wc ls-tree
0 232525 11611220 ls-tree
Could this cause some troubles to git-annex ? Maybe my repository is just too big and it demands some time to deal with the whole thing?
Before doing the git annex repair command I have tried to delete a distant annex (hosted on a raspberry pi, running an arm raspbian) from this one, using the git annex webapp website. Since it was kind of stuck I asked webapp to shutdown. I ran git annex repair for some time. It looked like stuck. I stopped it, then tried git fsck. It ran well. I tried git annex fsck, it took all my RAM and swap. The process was eating 14 Giga bytes of memory. I tried git gc and the same memory problem occured. I ran git annex repair again and it has been running for 38 hours now. I am lost. Please help.
vincent@berlioz:~$ LANG=C apt-cache policy git-annex
git-annex:
Installed: 5.20141125
Candidate: 5.20141125
Package pin: 5.20141125
Version table:
*** 5.20141125 950
400 http://ftp.fr.debian.org/debian/ testing/main amd64 Packages
50 http://ftp.fr.debian.org/debian/ unstable/main amd64 Packages
100 /var/lib/dpkg/status
3.20120629 950
900 http://ftp.fr.debian.org/debian/ wheezy/main amd64 Packages
vincent@berlioz:~/photos2$ du -sh .
152G .
Info gathered from another client annex:
vincent@tigrou:/mnt/mars/images/photos2$ git annex info
repository mode: indirect
trusted repositories: 0
semitrusted repositories: 11
8<------------skipped repositories list------------->8
untrusted repositories: 1
a357bb9e-7a9d-4112-b74c-13707e2c7f85 -- vincent@rhett:~/photos [gateway]
transfers in progress: none
available local disk space: 945.91 gigabytes (+1 megabyte reserved)
bad keys size: 368 bytes (clean up with git-annex unused)
local annex keys: 76070
local annex size: 383.91 gigabytes
annexed files in working tree: 59117
size of annexed files in working tree: 319.73 gigabytes
bloom filter size: 16 mebibytes (15.2% full)
backend usage:
SHA256E: 101902
SHA256: 33285
It seems I could sort things of.
I have run git annex sync from another annex. It took 30 minutes but could synchronize with the slow annex. Then I launched git annex sync on the slow annex (the one which seemed to had troubles with the repair command). It took 278 minutes, used all my ram and swap, but managed to end successfully.
I launched git annex sync again today, another 279 minutes:
I will try to do some git gc.
vincent@berlioz:~/photos2$ git config --global pack.packSizeLimit 100m vincent@berlioz:~/photos2$ git config --global pack.windowMemory 100m vincent@berlioz:~/photos2$ git config --global pack.threads 1 vincent@berlioz:~/photos2$ time git gc error: pack-objects died of signal 9 error: failed to run repack
real 21m29.451s user 1m31.414s sys 0m9.141s
According to the ls-tree output you posted, the git tree object 5fe5193e079631c0ceac0688ae0a6c1636491b61 is somehow 11 mb in size, which seems quite enormous, since a git tree object is just a list of files in a directory in the repo.
Does your repository actually contain a directory that big? (This seems unlikely, since git annex info says there are only 59117 annxed files.)
What is in the output of
git-ls-tree 5fe5193e079631c0ceac0688ae0a6c1636491b61that is so enormous?I think you need to figure this out to get anywhere, because it's quite strange. Or, if you're willing to put the git repository someplace (minus .git/annex/), I could take a look at it.
Thanks for your time and help.
http://hobeika.fr/~vincent/git/ls-tree
I'll have to find some place to store that.
Yet I don't think my repository holds such a big directory since there are no problems on other annexes.
The output of the ls-tree command suggests this tree is the git-annex branch at some point in time.
I tried on my photos annex to ls-tree the git-annex branch and got.
My git-annex tree is about 22Mib in size, that means twice bigger than yours (no dumb joke intended) and I don't suffer such problem.
I have created a 100 GB swap file and ran git gc again. It ate my 8 GB of RAM + 38 GB of swap. It ended successfully:
Yet the following git annex sync didn't do better than previous ones and took 285 minutes to complete.
I am going to check my hard disk drive to see if there aren't any third party problem.
Yet it shrank by 300 MB.
Hi,
I have uploaded the repository to: http://dl.free.fr/bczxhyOhy
You can decrypt it with your gpg key. It weights 200 Mo.
Moreover when I try to run git annex whereis on files present only on the defective annex from another annex I get the following error:
I guess the defective annex could not completely synchronize with the others.
I also tried to look at the git tree with gitk --all. Well there seems to be a lot of commit with message update (recovery from race) I am forced to kill gitk because it starts lagging a lot.
Hopefully you will be able to provide me some hints on how to resolve this issue.
Best regards
vho
This one should work: https://www.dropbox.com/s/t7b406wm6m3vm6c/photos2.tar.xz.gpg?dl=0
Finally got back to this. I downloaded the file.
You may be able to fix your repository by running
git annex forgetI guess this is the same problem described in git-annex branch shows commit with looong commitlog
Sorry I didn't see your answer before!
Thanks! Fortunately I kept a copy of the defective repository. I am going to test this =)