I do the following:
# create repo
git init repo
git -C ./repo annex init --version=6
git -C ./repo config annex.thin true
# add big file
cd repo
cp /tmp/bigfile . # 12GB file on system with 8GB RAM
git annex add bigfile
# commit
git commit -m 'msg'
On the commit, I get "fatal: Out of memory, realloc failed". I found out that "git add" leads to the same error with big files, so I use "git annex add".
I do this on an NFS4 disk. The same on the local EXT4 disk works.
I have noticed that annex creates a link for the file on EXT4 but not on NFS4. This seems suspicious.
Version information:
git version
git version 2.16.2
git annex version
build flags: Assistant Webapp Pairing S3(multipartupload)(storageclasses) WebDAV Inotify DBus DesktopNotify ConcurrentOutput TorrentParser MagicMime Feeds Testsuite
dependency versions: aws-0.16 bloomfilter-2.0.1.0 cryptonite-0.23 DAV-1.3.1 feed-0.3.12.0 ghc-8.0.2 http-client-0.5.7.1 persistent-sqlite-2.6.3 torrent-10000.1.1 uuid-1.3.13 yesod-1.4.5
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL
remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar hook external
local repository version: 6
supported repository versions: 3 5 6
upgrade supported from repository versions: 0 1 2 3 4 5
operating system: linux x86_64
I wanted to try it with a repository using v5 indirect mode. This tells me:
Is that the case with NFS4 in general or might the disk not be mounted properly?
Actually, I am still interested in a response.
I would prefer to use annex.addunlocked = true, which would use hardlinks.
Yes, see Joey's notes at smudge “When git add is run with a large file, it allocates memory for the whole file content, even though it's only going to stream it to the clean filter. My proposed smudge/clean interface patch also fixed this problem, since it made git not read the file at all.” I believe that
git annex addwill avoid this issue as you have seen.See also unlocked files:
git config annex.addunlocked truetells git annex to add all files as unlockedgit config annex.thin truetells git annex to only keep one copy of unlocked files by using hardlinksI believe both of these settings only have an effect on a V6 repo. I assume for your desires you will want to use both settings. Did you try setting both of these on your V6 repo then doing a
git annex add?The unlocked page and smudge page also talk about various limitations of the thin setting, for example from the smudge page: “…with annex.thin no attempt is made to protect the object from being modified. If a user wants to protect object contents from modification, they should use git annex add, not git add, or they can git annex lock after adding, or not enable annex.thin.”