$> git clone http://data.pymvpa.org/datasets/tutorial_data/.git tutorial_data-2
Cloning into 'tutorial_data-2'...
Checking connectivity... done.
cd
#35 !280 [0].....................................:Thu Jul 14 10:16:49:.
anthill:/home/ironfs/scratch/yarik
$> cd tutorial_data-2
total 36
4 freesurfer/ 4 hyperalignment_tutorial_data.hdf5.gz@ 20 suma_surfaces/
4 haxby2001/ 4 results/
(git)/home/ironfs/scratch/yarik/tutorial_data-2:[master]
#36 !281 [0].....................................:Thu Jul 14 10:16:50:.
anthill:/home/ironfs/scratch/yarik/tutorial_data-2
$> git annex get hyperalignment_tutorial_data.hdf5.gz
Detected a filesystem without POSIX fcntl lock support.
Enabling annex.pidlock.
(merging origin/git-annex into git-annex...)
(recording state in git...)
/home/ironfs/scratch 100%[===================>] 15.04M 68.9MB/s in 0.2s
(checksum...) ok
(recording state in git...)
(git)/home/ironfs/scratch/yarik/tutorial_data-2:[master]
#37 !282 [0].....................................:Thu Jul 14 10:16:54:.
anthill:/home/ironfs/scratch/yarik/tutorial_data-2
$> git annex drop hyperalignment_tutorial_data.hdf5.gz
drop hyperalignment_tutorial_data.hdf5.gz (checking origin...) git-annex: SQLite3 returned ErrorIO while attempting to perform prepare "PRAGMA journal_mode=WAL;": disk I/O error
(git)/home/ironfs/scratch/yarik/tutorial_data-2:[master]
$> git annex fsck
fsck freesurfer/anat_nii.nii sqlite worker thread crashed: SQLite3 returned ErrorIO while attempting to perform prepare "SELECT null from content limit 1": disk I/O error
git-annex: sqlite query crashed
$> git annex version
git-annex version: 6.20160613-g35dbe35
build flags: Assistant Webapp Pairing Testsuite S3(multipartupload)(storageclasses) WebDAV Inotify DBus DesktopNotify XMPP ConcurrentOutput TorrentParser MagicMime Feeds Quvi
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 SHA1E SHA1 MD5E MD5 WORM URL
remote types: git gcrypt S3 bup directory rsync web bittorrent webdav tahoe glacier ddar hook external
local repository version: 5
supported repository versions: 5 6
upgrade supported from repository versions: 0 1 2 3 4 5
operating system: linux x86_64
(git)/home/ironfs/scratch/yarik/tutorial_data:[master]
The v6 changes added a sqlite database. Some code will try to query or write that database even in v5 mode, although it's meant to give up if the database is not available.
So, the easy fix, at least for the drop problem, is to avoid using the keys database at all in v5 mode. I've made this change and it will probably fix the case you reported.
But, that won't help with v6 repos which need to use that sqlite database. And, incremental fsck uses its own sqlite database too. And, caching database plans are to use sqlite databases more broadly in the future.
I'm sure that it's not a good idea for git-annex to catch "disk IO error" exceptions from the database layer. So, it seems that most any other fix than avoiding using the database would need to be made in sqlite or in lustre, which it seems don't get along. At a guess, sqlite is trying to use some POSIX filesystem functionality, likely related to locking, that lustre does not support.
Hmm, what could be done to hack in support for lustre is to move the sqlite databases to a different filesystem. But, accessing the same repo from different hosts which have different sqlite databases would lead to inconsistent and buggy behavior. And repo setup would need to decide where to put the sqlite databases and manually configure that location. So this would be very much a caveat empror configuration.
It's now possible to set annex.dbdir to a directory on some other filesystem than one containing the repository, where sqlite works better.
Unsure yet if it would be safe to set annex.dbdir to a local directory, and have the git-annex repository on a networked filesystem that is accessed from multiple clients. Each client will have its own set of sqlite databases, and will maintain them as best it can.