Recent comments posted to this site:
Forgot to add in the previous comment. The index looks fine afterwards
On branch master
No commits yet
Changes to be committed:
(use "git rm --cached <file>..." to unstage)
new file: a
new file: b
new file: c
diff --git a/a b/a
new file mode 100755
index 0000000..f8e47b9
--- /dev/null
+++ b/a
@@ -0,0 +1 @@
+/annex/objects/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
diff --git a/b b/b
new file mode 100755
index 0000000..f8e47b9
--- /dev/null
+++ b/b
@@ -0,0 +1 @@
+/annex/objects/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
diff --git a/c b/c
new file mode 100755
index 0000000..f8e47b9
--- /dev/null
+++ b/c
@@ -0,0 +1 @@
+/annex/objects/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
I found a new type of failure which occurs when there are new unlocked files in the index.
git init
git annex init
git config annex.crippledfilesystem true
git config annex.addunlocked true
touch a
git annex add . # OK
touch b
git annex add . # 1 error
touch c
git annex add . # 2 errors
Something is happening to the files already in the index and the error is triggered once per file in the index.
add c
ok
sqlite worker thread crashed: user error (SQLite3 returned ErrorProtocol while attempting to perform prepare "SELECT null from content limit 1": locking protocol(while opening database connection))
git-annex: sqlite query crashed: thread blocked indefinitely in an MVar operation
CallStack (from HasCallStack):
error, called at ./Database/Handle.hs:102:40 in main:Database.Handle
error: external filter 'git-annex smudge --clean -- %f' failed 1
error: external filter 'git-annex smudge --clean -- %f' failed
add a
ok
sqlite worker thread crashed: user error (SQLite3 returned ErrorProtocol while attempting to perform prepare "SELECT null from content limit 1": locking protocol(while opening database connection))
git-annex: sqlite query crashed: thread blocked indefinitely in an MVar operation
CallStack (from HasCallStack):
error, called at ./Database/Handle.hs:102:40 in main:Database.Handle
error: external filter 'git-annex smudge --clean -- %f' failed 1
error: external filter 'git-annex smudge --clean -- %f' failed
add b
ok
(recording state in git...)
The error I get previously (before 0f38ad9a6) with my test case is
init
Detected a filesystem without fifo support.
Disabling ssh connection caching.
ok
(recording state in git...)
get a (from origin...)
ok
get b (from origin...)
ok
(recording state in git...)
sqlite worker thread crashed: user error (SQLite3 returned ErrorProtocol while attempting to perform prepare "SELECT null from content limit 1": locking protocol(while opening database connection))
git-annex: sqlite query crashed: thread blocked indefinitely in an MVar operation
CallStack (from HasCallStack):
error, called at ./Database/Handle.hs:102:40 in main:Database.Handle
error: external filter 'git-annex smudge --clean -- %f' failed 1
error: external filter 'git-annex smudge --clean -- %f' failed
sqlite worker thread crashed: user error (SQLite3 returned ErrorProtocol while attempting to perform prepare "SELECT null from content limit 1": locking protocol(while opening database connection))
git-annex: sqlite query crashed: thread blocked indefinitely in an MVar operation
CallStack (from HasCallStack):
error, called at ./Database/Handle.hs:102:40 in main:Database.Handle
error: external filter 'git-annex smudge --clean -- %f' failed 1
error: external filter 'git-annex smudge --clean -- %f' failed
With d0ef8303c, the test case still works, but adjusted branches still have the same error.
git init
git annex init
git config annex.crippledfilesystem true
echo aaa > a
cp a b
git annex add .
git commit -m .
git annex adjust --unlock
produces
adjust
sqlite worker thread crashed: user error (SQLite3 returned ErrorProtocol while attempting to perform prepare "SELECT null from content limit 1": locking protocol(while opening database connection))
git-annex: thread blocked indefinitely in an MVar operation
error: external filter 'git-annex smudge -- %f' failed 1
error: external filter 'git-annex smudge -- %f' failed
sqlite worker thread crashed: user error (SQLite3 returned ErrorProtocol while attempting to perform prepare "SELECT null from content limit 1": locking protocol(while opening database connection))
git-annex: thread blocked indefinitely in an MVar operation
error: external filter 'git-annex smudge -- %f' failed 1
error: external filter 'git-annex smudge -- %f' failed
Switched to branch 'adjusted/master(unlocked)'
sqlite worker thread crashed: user error (SQLite3 returned ErrorProtocol while attempting to perform prepare "SELECT null from content limit 1": locking protocol(while opening database connection))
git-annex: sqlite query crashed: thread blocked indefinitely in an MVar operation
CallStack (from HasCallStack):
error, called at ./Database/Handle.hs:79:40 in main:Database.Handle
failed
adjust: 1 failed
About git-annex version, I'm using make install-home to do an incremental build but the version does not update.
Since the vast number of places git-annex runs do not have this problem, I don't think we can draw any conclusions about differences between your 2 machines.
Memory issue still seems like the most likely bet.
Are you able to reproduce the problem repeatedly?
I have exactly the same problem with Debian 11 and git-annex 8.20210223. In kernel.log I can see
traps: git-annex[91341] general protection fault ip:3c6be67 sp:7ffd7afd26f8 error:0 in git-annex[400000+3a78000]
I've done a memtest86 on my memory and nothing appended.
Oddly, I have another machin with same versions of OS and git-annex, but it does not reproduce the problem. The bigger difference between the two machines is that machine with problem run on ZFS. Is it possible that the problem comes from that?
Well, I tried to reproduce this, following your instructions to the extent there were clear.
I made a repository, added some files to git, and committed.
Then git rm --cached the files, and git-annex add to add them to
git-annex, and committed. So master had 2 commits, an older commit
where the files were in git, and a newer commit where the files are in
git-annex. The git-annex branch had a couple of commits as well.
Then I cloned:
git clone --depth 1 localhost:/tmp/path/to/repo clone
In the clone, that made master be a single commit. There was no origin/git-annex branch in the clone, like there normally would be in a non-shallow clone.
Then I ran git-annex init; git annex sync:
joey@darkstar:/tmp/clone>git annex init
init ok
(recording state in git...)
joey@darkstar:/tmp/clone>git annex sync
commit
On branch master
Your branch is up to date with 'origin/master'.
nothing to commit, working tree clean
ok
pull origin
ok
push origin
Enumerating objects: 6, done.
Counting objects: 100% (6/6), done.
Delta compression using up to 4 threads
Compressing objects: 100% (3/3), done.
Writing objects: 100% (5/5), 450 bytes | 450.00 KiB/s, done.
Total 5 (delta 0), reused 0 (delta 0), pack-reused 0
To localhost:/tmp/repo
* [new branch] master -> synced/master
* [new branch] git-annex -> synced/git-annex
ok
At this point, the git-annex branch in the remote repository is not destroyed. It contains a merge between the branch that was there before the clone and the git-annex branch that was synced from the clone. Looks just fine.
In the clone, there is still no origin/git-annex branch, and the git-annex branch has only the changes that git-annex committed to it in the clone.
So, the clone still doesn't know that it can get the annexed files from origin. But nothing is "destroyed".
This does not seem like the best possible behavior, it would be better if, after git-annex sync, it fetched origin/git-annex (either the latest commit or all of them) and merged it into the local git-annex branch.
What's going on is, the shallow clone gets remote.origin.fetch set to "+refs/heads/master:refs/remotes/origin/master". So, attempting to fetch any other branch from origin will always skip creating a tracking branch.
All you have to do, then is
git config '+refs/heads/*:refs/remotes/origin/*'
That preserves master as a shallow clone, while letting
the git-annex branch be fetched. Or, alternatively, git fetch --unshallow.
Maybe git-annex sync could detect this situation and force fetch
the git-annex branch. (eg, git fetch origin git-annex, which does
actually fetch the refs, followed by manually setting
origin/git-annex to FETCH_HEAD) That would leave workflows using
git push and git pull still with the problem. And it might be that
someone who wants a shallow clone also wants the git-annex branch to be
cloned shallowly and would object if its full history was fetched by that.
I have not found a way yet to fetch the git-annex branch shallowly.
If your filesystem supports reflinks, you should not need to enable
annex.thin, just let git-annex make copies. It makes the copy
using cp --reflink=auto, so when reflinks are supported, you'll get a
nice cheap reflink.
WRT annex.thinmode=forcehardlink, this would be something that aimed the gun right at the user's foot and then waits for the trigger to be pulled by any program that ever might write to a file.
This is not limited to rsync.net, other DNS lookups by git-annex also fail.
Without investigating, my strong hunch is that this is because git-annex is linked to glibc, and so it expects there to be /etc/nssswitch.conf, /etc/resolv.conf, /etc/services etc that glibc uses. And some/all of those files are not present on Android. If you were able to set up the files, it would probably work.
Yes the problem occurs each time I want to do git annex get . on a repository with about 20 000 files.
As i say, I've perfom memtest86 (several times) and memtester (several times) without any problem. Do you think it could be hardware problem despite these results ?
You suggest using gdb. I get
Thread 1 "git-annex" received signal SIGSEGV, Segmentation fault. 0x0000000003c6be67 in ?? ()
(gdb) backtrace
0 0x0000000003c6be67 in ?? ()
1 0x0000000000000000 in ?? ()
which is not very helpful. Do you have any advices to investigate?