Please describe the problem.
When I copy file from on directory to another, I have the following error
$ git annex copy . --to another
...
copy file1 (to another...)
100% 321.93 MiB 115 MiB/s 0ssqlite worker thread crashed: SQLite3 returned ErrorCan'tOpen while attempting to perform open "repo/.git/annex/keysdb/db".
sqlite query crashed: thread blocked indefinitely in an MVar operation
CallStack (from HasCallStack):
error, called at ./Database/Handle.hs:102:40 in main:Database.Handle
transfer failed
ok
copy file2 (to another...)
100% 424.24 MiB 109 MiB/s 0ssqlite worker thread crashed: SQLite3 returned ErrorCan'tOpen while attempting to perform open "repo/.git/annex/keysdb/db".
sqlite query crashed: thread blocked indefinitely in an MVar operation
CallStack (from HasCallStack):
error, called at ./Database/Handle.hs:102:40 in main:Database.Handle
transfer failed
ok
...
What steps will reproduce the problem?
Any time I copy files to this remote
What version of git-annex are you using? On what operating system?
On local machine:
$ git-annex version
git-annex version: 8.20210223
build flags: Assistant Webapp Pairing Inotify DBus DesktopNotify TorrentParser MagicMime Feeds Testsuite S3 WebDAV
dependency versions: aws-0.22 bloomfilter-2.0.1.0 cryptonite-0.26 DAV-1.3.4 feed-1.3.0.1 ghc-8.8.4 http-client-0.6.4.1 persistent-sqlite-2.10.6.2 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.1.0
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL X*
remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs httpalso borg hook external
operating system: linux x86_64
supported repository versions: 8
upgrade supported from repository versions: 0 1 2 3 4 5 6 7
local repository version: 8
$ uname -a
Linux kiwi 5.10.0-19-amd64 #1 SMP Debian 5.10.149-2 (2022-10-21) x86_64 GNU/Linux
On remote machine:
$ git-annex version
git-annex version: 8.20200617
build flags: Assistant Webapp Pairing S3 WebDAV Kqueue DBus DesktopNotify TorrentParser MagicMime Feeds Testsuite
dependency versions: aws-0.22 bloomfilter-2.0.1.0 cryptonite-0.27 DAV-1.3.4 feed-1.3.0.1 ghc-8.10.3 http-client-0.7.1 persistent-sqlite-2.10.6.2 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.0.1
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL
remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs hook external
operating system: freebsd x86_64
supported repository versions: 8
upgrade supported from repository versions: 0 1 2 3 4 5 6 7
$ uname -a
FreeBSD extra-2 13.1-RELEASE-p2 FreeBSD 13.1-RELEASE-p2 n245412-484f039b1d0 TRUENAS amd64
Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
I use git-annex for several years, and I'm very happy with it. I's one of the best files synchronisation tools.
fixed, when git-annex is built against persistent-sqlite version 2.13.3. --Joey
Seems to me that you are not able to read the
repo/.git/annex/keysdb/dbfile, likely due to a permissions problem or because it is owned by another user.I was able to reproduce this by making the repository itself owned by me, but making that specific file in it owned by a different user and not readable by me:
Thank you for your anser
I've checked user right and all seem ok for me:
Thus for me the solution is somewhere else.
Hi,
Any idea explainig this failure?
Version 8.20210223 is quite old, it would be good to try with a newer version and see if the error message is different.
You might also try strace and look to see what it says happens when it tries to open "repo/.git/annex/keysdb/db".
I don't know if what you showed about file permissions means you can really read it. There might be ACLs or other complications? The best way to show you can read it is to actually try to read it, eg
cat db >/dev/nullHi, and thank you for keeping anwsering my question!
I do succeed in reading this file. And here is the content:
sqlite> .dump PRAGMA foreign_keys=OFF; BEGIN TRANSACTION; CREATE TABLE IF NOT EXISTS "associated"("id" INTEGER PRIMARY KEY,"key" BLOB NOT NULL,"file" BLOB NOT NULL,CONSTRAINT "key_file_index" UNIQUE ("key","file"),CONSTRAINT "file_key_index" UNIQUE ("file","key")); CREATE TABLE IF NOT EXISTS "content"("id" INTEGER PRIMARY KEY,"key" BLOB NOT NULL,"inodecache" VARCHAR NOT NULL,"filesize" VARCHAR NOT NULL,"mtime" INTEGER NOT NULL,CONSTRAINT "key_inode_cache_index" UNIQUE ("key","inodecache"),CONSTRAINT "inode_cache_key_index" UNIQUE ("inodecache","key")); COMMIT; sqlite>
Problem identified!
The directory
repowas actually containing "é" char. When I remove this char, problem disapeared.Oho, I reproduced it!
First I made a git-annex repo named "fooé", and added a file "foo" to it, and that is unlocked.
Note that using git-annex in repo fooé with LANG=C works. The problem seems limited to a remote with a unicode character in its name when not in a unicode locale.
In strace I see this:
That doesn't look like the correct encoding for "é" does it?
"/tmp/foo\303\251"would be correct and is what git-annex otherwise uses when accessing that repo.I think the reason for this is simply that persistent-sqlite uses Text for the location of the database. And Text is unicode encoded. So when the non unicode locale results in a FilePath that is encoded using the filesystem encoding, with surrogate characters, and that gets fed to Data.Text.pack, it replaces the "invalid scalar values" with "\65533".
The same thing would happen in a unicode locale if the remote's path was not valid unicode.
Filed an issue to get persistent-sqlite to not use Text for the FilePath https://github.com/yesodweb/persistent/issues/1481
I don't think that non-unicode FilePaths can be generally squeezed into a Text, so we may need to wait persistent getting fixed. Although it should be possible, in a non-unicode locale, to convert a non-unicode FilePath like "fooé" to a Text.
Also notice that git-annex succeeds despite this error. Which is reasonable since it was only unable to update the remote's keys db, and the remote can and will just update it itself next time git-annex is used over there. Which will work since that git-annex will be running in the directory and will use relative paths.
So perhaps the error message could just be suppressed?
I sent in a pull request today to persistent-sqlite that will let git-annex fix this once accepted.
https://github.com/yesodweb/persistent/pull/1486
Once/if that is merged, git-annex can be changed to use the new
open'