(Found the problem, so you might want to skip to the comments to see what's wrong.)

Please describe the problem.

Trying to drop the contents of 131 files in a subdirectory, but it dies with an "sqlite worker thread crashed" error. The contents of one file is deleted, and then it dies. The files in that directory are exactly 1G (1000000000) bytes each.

There's no data loss here, it creates files in .git/annex/journal/ which is added to the git-annex branch after a git annex sync. "git annex fsck --fast --quiet" is happy and finds no errors.

What steps will reproduce the problem?

"git annex drop" with and without --force. Tried to copy one of the files to another server with Debian 8.7, and it's successfully dropped there. Same git-annex version.

What version of git-annex are you using? On what operating system?

OS: Ubuntu 14.04.5 LTS

git-annex: 6.20161211-gc3ab3c668 (from git-annex-standalone-amd64.tar.gz in the annex at downloads.kitenet.net)

I have compiled the newest sqlite3 3.16.2 and placed it in /usr/local/bin/, just mentioning it because it's some kind of SQLite error. but executing the command with a PATH without /usr/local/bin makes no difference.

Please provide any additional information below.

If you can, paste a complete transcript of the problem occurring here.

If the problem is with the git-annex assistant, paste in .git/annex/daemon.log

$ git annex drop -d --force
[2017-01-16 08:58:39.947856861] read: git ["--git-dir=../../../../../../../.git","--work-tree=../../../../../../..","--literal-pathspecs","ls-files","--cached","-z","--"]
[2017-01-16 08:58:39.953150944] chat: git ["--git-dir=../../../../../../../.git","--work-tree=../../../../../../..","--literal-pathspecs","check-attr","-z","--stdin","annex.backend","annex.numcopies","annex.largefiles","--"]
[2017-01-16 08:58:39.953742873] read: git ["--version"]
[2017-01-16 08:58:39.959438053] process done ExitSuccess
[2017-01-16 08:58:39.959919735] read: git ["--git-dir=../../../../../../../.git","--work-tree=../../../../../../..","--literal-pathspecs","show-ref","git-annex"]
[2017-01-16 08:58:39.963707897] process done ExitSuccess
[2017-01-16 08:58:39.963882344] read: git ["--git-dir=../../../../../../../.git","--work-tree=../../../../../../..","--literal-pathspecs","show-ref","--hash","refs/heads/git-annex"]
[2017-01-16 08:58:39.967078602] process done ExitSuccess
[2017-01-16 08:58:39.969062554] chat: git ["--git-dir=../../../../../../../.git","--work-tree=../../../../../../..","--literal-pathspecs","cat-file","--batch"]
[2017-01-16 08:58:39.970168767] chat: git ["--git-dir=../../../../../../../.git","--work-tree=../../../../../../..","--literal-pathspecs","cat-file","--batch-check=%(objectname) %(objecttype) %(objectsize)"]
drop MyStream_3/MyStream_3.mp4.split_ag [2017-01-16 08:58:39.984359273] Dropping from here proof: Nothing
sqlite worker thread crashed: SQLite3 returned ErrorCan'tOpen while attempting to perform prepare "SELECT null from content limit 1": unable to open database file
ok
git-annex: thread blocked indefinitely in an MVar operation
$ git annex drop -d --force
[2017-01-16 08:58:58.890031175] read: git ["--git-dir=../../../../../../../.git","--work-tree=../../../../../../..","--literal-pathspecs","ls-files","--cached","-z","--"]
[2017-01-16 08:58:58.895047131] chat: git ["--git-dir=../../../../../../../.git","--work-tree=../../../../../../..","--literal-pathspecs","check-attr","-z","--stdin","annex.backend","annex.numcopies","annex.largefiles","--"]
[2017-01-16 08:58:58.895690334] read: git ["--version"]
[2017-01-16 08:58:58.901206493] process done ExitSuccess
[2017-01-16 08:58:58.901836453] read: git ["--git-dir=../../../../../../../.git","--work-tree=../../../../../../..","--literal-pathspecs","show-ref","git-annex"]
[2017-01-16 08:58:58.905632574] process done ExitSuccess
[2017-01-16 08:58:58.905793223] read: git ["--git-dir=../../../../../../../.git","--work-tree=../../../../../../..","--literal-pathspecs","show-ref","--hash","refs/heads/git-annex"]
[2017-01-16 08:58:58.909227509] process done ExitSuccess
[2017-01-16 08:58:58.910902084] chat: git ["--git-dir=../../../../../../../.git","--work-tree=../../../../../../..","--literal-pathspecs","cat-file","--batch"]
[2017-01-16 08:58:58.912164764] chat: git ["--git-dir=../../../../../../../.git","--work-tree=../../../../../../..","--literal-pathspecs","cat-file","--batch-check=%(objectname) %(objecttype) %(objectsize)"]
drop MyStream_3/MyStream_3.mp4.split_ah [2017-01-16 08:58:58.927477693] Dropping from here proof: Nothing
sqlite worker thread crashed: SQLite3 returned ErrorCan'tOpen while attempting to perform prepare "SELECT null from content limit 1": unable to open database file
ok
git-annex: thread blocked indefinitely in an MVar operation
$

End of transcript or log.

Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)

Indeed. All my stuff (around 3.5 terabytes) is stored in git-annex with at least three copies of each file on different disks and locations, spread over various hard disks, memory sticks, servers and you name it. Unused disk space is a waste, so I fill everything up to the brim with extra copies.

In other words, Git-Annex and I are very happy together, and I'd like to marry it. And because you are the father, I hereby respectfully ask for your blessing.

fixed (and I suppose you have my blessing, but I'm not sure that's legal yet!) --Joey

RSS Atom

Found the reason

I managed to find the reason why it failed. The files in .git/annex/keys/ had wrong permissions, so I wasn't able to write to them. This is a shared repository where everyone within the Unix group can read and write. All the directories including .git/annex/keys have chmod +s so any additions or edits are stored within the correct group. But the SQLite files were stored with permission 0644 instead of 0664, and my umask in the shell is 0002. Maybe SQLite is creating the database with wrong permissions?

Comment by sunny256 — Mon Jan 16 11:03:45 2017

Remove comment

Wrong permissions

Yes, confirmed. I tried to create a new test repository and chmod all directories with +s. I'm using umask 0002, so new files are created with perm 0664 and the correct shared group. But the SQLite files in .git/annex/keys/ are created with permission 0644, so other users in the group are not able to update the files there.

Comment by sunny256 — Mon Jan 16 11:17:31 2017

Remove comment

comment 3

Thanks for following up with the cause of this.

In fact, assuming you're not using a v6 git-annex repository, it doesn't really need to update that database at all. But since we'll be upgrading to v6 eventually, I need to deal with problems like this. Also, this same problem will also impact the database used for incremental fsck.

I can reproduce this with a v5 repository; dropping a file happens to run a code path that updates the database. And reproducing it w/o using git-annex too:

joey@darkstar:~/tmp>mkdir empty
joey@darkstar:~/tmp>umask
0002
joey@darkstar:~/tmp>touch empty/file
joey@darkstar:~/tmp>sqlite3 empty/db
SQLite version 3.16.2 2017-01-06 16:32:41
Enter ".help" for usage hints.
sqlite> create table foo;
Error: near ";": syntax error
sqlite> 
joey@darkstar:~/tmp>ls -l empty/
total 0
-rw-r--r-- 1 joey joey 0 Feb 13 16:33 db
-rw-rw-r-- 1 joey joey 0 Feb 13 16:32 file

Seems that sqlite uses 0644 & umask for the db permissions, which is bad because it doesn't allow the umask to enable the group write bit. That 0644 is SQLITE_DEFAULT_FILE_PERMISSIONS, so it can be changed to something saner at compile time.

http://www.sqlite.org/src/doc/trunk/src/os_unix.c has a useful comment. Seems that sqlite is careful to make -wal, -journal, and -shm files have the exact same permissions as main database file.

So, if .git/annex/keys/* is updated to have the desired permissions when the database is created, every further write to the database will keep using the desired permissions.

Hmm, it turns out that git-annex does already set the database permissions when creating it, but only if core.sharedRepository is set to group or all. So there's a workaround; just git config core.sharedRepository group when setting up a repository that's going to be accessed by multiple users. Almost certianly a better idea than relying on umask anyway; people mess up umask settings.

I'll go ahead and make it always set sane permissions when creating databases. I'm not going to try to fix up permissions in existing repositories though.

Comment by joey — Mon Feb 13 20:21:09 2017

Remove comment

Add a comment