forum/Out of memory on NFS4 with files bigger than available memorygit-annexhttp://git-annex.branchable.com/forum/Out_of_memory_on_NFS4_with_files_bigger_than_available_memory/git-annexikiwiki2018-07-09T13:44:15Zcomment 1http://git-annex.branchable.com/forum/Out_of_memory_on_NFS4_with_files_bigger_than_available_memory/comment_1_2e89bcc3fbd069e5d9eb12c1c19993ad/yves.noirjean2018-06-08T15:17:16Z2018-06-08T15:17:16Z
<p>I wanted to try it with a repository using v5 indirect mode. This tells me:</p>
<pre><code> Filesystem allows writing to files whose write bit is not set.
git-annex: This repository seems to be on a crippled filesystem, you must use direct mode.
</code></pre>
<p>Is that the case with NFS4 in general or might the disk not be mounted properly?</p>
comment 2http://git-annex.branchable.com/forum/Out_of_memory_on_NFS4_with_files_bigger_than_available_memory/comment_2_cf8065e697a0a83cf3461672cb6421e0/yves.noirjean2018-06-18T11:25:18Z2018-06-18T11:25:18Z
The issue has been resolved. The unix file permissions were not applied, but the NFS4 permissions. So git annex did not trust that files without write flags could not be changed. Therefore, it didn't move the files and create symlinks. This somehow lead to git allocating a lot of memory during commit.
comment 3http://git-annex.branchable.com/forum/Out_of_memory_on_NFS4_with_files_bigger_than_available_memory/comment_3_e6bd4eb070a094116e13064dca6d5bb5/yves.noirjean2018-06-21T16:15:05Z2018-06-21T16:15:05Z
<p>Actually, I am still interested in a response.</p>
<p>I would prefer to use annex.addunlocked = true, which would use hardlinks.</p>
comment 4http://git-annex.branchable.com/forum/Out_of_memory_on_NFS4_with_files_bigger_than_available_memory/comment_4_42adc29a2c3216910a3bcda04d6f6a6b/andrew2018-07-09T13:44:15Z2018-07-09T13:44:15Z
<p>Yes, see Joey's notes at <a href="http://git-annex.branchable.com/todo/smudge/">smudge</a> <em>“When git add is run with a large file, it allocates memory for the whole file content, even though it's only going to stream it to the clean filter. My proposed smudge/clean interface patch also fixed this problem, since it made git not read the file at all.”</em> I believe that <code>git annex add</code> will avoid this issue as you have seen.</p>
<p>See also <a href="http://git-annex.branchable.com/tips/unlocked_files/">unlocked files</a>:</p>
<ul>
<li> <code>git config annex.addunlocked true</code> tells git annex to add all files as unlocked</li>
<li> <code>git config annex.thin true</code> tells git annex to only keep one copy of unlocked files by using hardlinks</li>
</ul>
<p>I believe both of these settings only have an effect on a V6 repo. I assume for your desires you will want to use both settings. Did you try setting both of these on your V6 repo then doing a <code>git annex add</code>?</p>
<p>The unlocked page and smudge page also talk about various limitations of the thin setting, for example from the smudge page: <em>“…with annex.thin no attempt is made to protect the object from being modified. If a user wants to protect object contents from modification, they should use git annex add, not git add, or they can git annex lock after adding, or not enable annex.thin.”</em></p>