cryptonite does not use SHA instructions for sha256git-annexhttp://git-annex.branchable.com/bugs/git-annex_is_slow_at_reading_file_content/git-annexikiwiki2024-03-11T14:00:49Zcomment 1http://git-annex.branchable.com/bugs/git-annex_is_slow_at_reading_file_content/comment_1_ed54af250541e2add89938b939016659/Atemu2022-03-04T14:16:53Z2022-03-04T14:16:53Z
<p>To expand on this, results from my more powerful machine reading a 14G file (all in tmpfs):</p>
<table>
<thead>
<tr>
<th>Command</th>
<th>Time</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>git-annex fsck file</code></td>
<td> 45s </td>
</tr>
<tr>
<td><code>sha256sum file</code></td>
<td> 7s </td>
</tr>
<tr>
<td><code>cat file > /dev/null</code></td>
<td> 1s </td>
</tr>
</tbody>
</table>
<p>(5800X with dual channel quad rank 3600MT/s CL16 RAM)</p>
comment 2http://git-annex.branchable.com/bugs/git-annex_is_slow_at_reading_file_content/comment_2_786b21544854ad719eac07d08ba8d206/Atemu2022-03-04T14:44:29Z2022-03-04T14:44:29Z
<p>(Sorry, command used on the more powerful machine was <code>git annex add</code>, not <code>fsck</code>.)</p>
<p>The read speed here (~320MiB/s) is pretty close to the <code>openssl speed sha256</code> result for 16B block size (~360MiB/s) or 64B blocks with SHA256 acceleration disabled (~300MiB/s, <code>OPENSSL_ia32cap=:~0x20000000</code>).</p>
<p>What block size is used here and does git-annex use SHA instructions for hashing?</p>
comment 3http://git-annex.branchable.com/bugs/git-annex_is_slow_at_reading_file_content/comment_3_190a153449f7bb07fb4aca89db236881/joey2022-03-07T19:30:28Z2022-03-07T19:09:17Z
<p>Cannot reproduce that:</p>
<pre><code>joey@darkstar:/tmp/rr>/usr/bin/time git-annex fsck bigfile
fsck bigfile (checksum...) ok
(recording state in git...)
10.08user 0.35system 0:10.40elapsed 100%CPU (0avgtext+0avgdata 43500maxresident)k
joey@darkstar:/tmp/rr>/usr/bin/time sha256sum bigfile
7219108977be820830169fc13fded5000e9db329fab4e687fc14236f932d4060 bigfile
11.00user 0.28system 0:11.29elapsed 99%CPU (0avgtext+0avgdata 1956maxresident)k
120inputs+0outputs (2major+105minor)pagefaults 0swaps
</code></pre>
<p>(1 gb file; tmpfs)</p>
<p>git-annex simply uses
<a href="https://hackage.haskell.org/package/cryptonite">cryptonite</a> for hashing.
Which is a thin wrapper around C code, and contains optimised routines for
some processors, generally copied from the reference C implementations.</p>
<p>The block size is 32 kb. This can be changed in Utility/Metered.hs
in defaultChunkSize.</p>
comment 4http://git-annex.branchable.com/bugs/git-annex_is_slow_at_reading_file_content/comment_4_492a11bcabc20b02239bc3b12438a51a/Atemu2022-03-08T11:56:42Z2022-03-08T11:56:42Z
<p>I found the issue, cryptonite doesn't make use of SHA instructions which both of my machines have. I'm going to open an issue upstream.</p>
<p>Is there an alternative crypto library that can make use of these perhaps? The speedup/efficiency gain is <strong>>5x</strong> on my 5800X.</p>
comment 5http://git-annex.branchable.com/bugs/git-annex_is_slow_at_reading_file_content/comment_5_1367e6e461a9a049527197c262635c2f/Atemu2022-03-08T13:24:21Z2022-03-08T13:24:21Z
Upstream bug report: https://github.com/haskell-crypto/cryptonite/issues/361
comment 6http://git-annex.branchable.com/bugs/git-annex_is_slow_at_reading_file_content/comment_6_123c81ad10aef15814f1c831e18c0a23/joey2022-03-08T17:08:29Z2022-03-08T16:26:55Z
<p>Awesome that you were able to reproduce it in cryptonite and file that.
Thank you!</p>
<p>I'm inclined to close this, because it seems unlikely there will be a fix
for it in git-annex. One fix that would be possible here is for
git-annex to use the exteneral sha256sum command. But, that would prevent
git-annex from displaying progress while hashing, which is why it moved
away from it in the first place.</p>
<p>Of course, you could also try using a different kind of hash that is
faster for new content.</p>
comment 7http://git-annex.branchable.com/bugs/git-annex_is_slow_at_reading_file_content/comment_7_2885c2f164dbe59234568966e901bdd7/joey2022-03-08T17:54:18Z2022-03-08T17:47:37Z
<p>I don't know of anything else in the haskell space that is likely to be
better, cryptonite is basically the standard library for this. There is
certianly precident in it for using optimised instructions, like SSE.</p>
comment 8http://git-annex.branchable.com/bugs/git-annex_is_slow_at_reading_file_content/comment_8_559e5c3b5ea9c166b3aa4a2e975643c6/Atemu2022-03-09T11:40:28Z2022-03-09T11:40:28Z
<p>SHA256 is by far the fastest hash on my Celeron J4105 with acceleration. Seconds is BLAKE at around half the speed of accelerated SHA256 or over twice the speed of unaccelerated SHA256.</p>
<p>Seems like coreutils simply uses openssl when available and openssl handles HW acceleration. I just tried using two Haskell OpenSSL wrappers (hopenssl and HsOpenSSL) in my minimal example and it was actually <em>faster</em> than the openssl CLI utility by almost 100MiB/ (though still a bit slower than <code>sha256sum</code>).</p>
<p>Would using those be an option?</p>
comment 9http://git-annex.branchable.com/bugs/git-annex_is_slow_at_reading_file_content/comment_9_c8b4c2e955b4ef5caaaaf2f834fb37f8/Atemu2022-05-25T12:22:58Z2022-05-25T12:22:58Z
<p>Upstream does not seem interested in improving performance. Are there alternative libraries that could be used instead?</p>
<p>Would it be possible to use OpenSSL somehow perhaps?</p>
<p>AFAICT git-annex hooks into the hashing routine for things like progress monitoring. Perhaps those could be re-architected to work with an external "black box" library or process?</p>
comment 10http://git-annex.branchable.com/bugs/git-annex_is_slow_at_reading_file_content/comment_10_d04b5bc031f73363d44b81f436b59781/joey2023-09-22T16:58:21Z2023-09-22T16:57:14Z
<p>crytonite has been forked to crypton, which git-annex can build with as an
option. It may be that the maintainer of that
will be more open to this kind of optimisation.
<a href="https://github.com/kazu-yamamoto/crypton/issues">https://github.com/kazu-yamamoto/crypton/issues</a></p>
comment 11http://git-annex.branchable.com/bugs/git-annex_is_slow_at_reading_file_content/comment_11_cab37d851237a7798f74867e08aaafb9/joey2024-03-01T20:53:17Z2024-03-01T18:53:39Z
<p>My laptop now has SHA256 in hardware I assume, as I'm seeing similar speed
differences. Eg:</p>
<pre><code>joey@darkstar:~/tmp/x3>time sha256sum x
e1f21b651362cf2f86ccc14a7376c33a49f30d7d14afe1ec6993eead5e8cfe41 x
0.53user 0.17system 0:00.71elapsed 99%CPU (0avgtext+0avgdata 3712maxresident)k
32inputs+0outputs (1major+239minor)pagefaults 0swaps
joey@darkstar:~/tmp/x3>time git-annex fsck x
fsck x (checksum...) ok
(recording state in git...)
3.22user 0.40system 0:03.63elapsed 100%CPU (0avgtext+0avgdata 61180maxresident)k
0inputs+144outputs (0major+8547minor)pagefaults 0swaps
</code></pre>
<p>I agree that this bug should be left open since even a relatively low-end laptop
now has this, git-annex shouldn't leave so much performance on the table.</p>
<p>I've opened an issue on crypton:
<a href="https://github.com/kazu-yamamoto/crypton/issues/31">https://github.com/kazu-yamamoto/crypton/issues/31</a></p>
<p>If it's rejected from crypton, I'm open to considering another library.
I'd rather avoid OpenSSL wrappers because adding a C library dependency on openssl
will complicate building git-annex in some situations. It would be better to
have a haskell library that, like cryptonite, embeds the necessary C code.</p>
<p>Also, whatever library git-annex uses needs to support incremental hashing,
otherwise git-annex has to pay a performance penalty of re-reading a file to hash
it after download, rather than hashing while downloading.</p>
comment 12http://git-annex.branchable.com/bugs/git-annex_is_slow_at_reading_file_content/comment_12_466a365bca48065cc6b9e280b5c385a5/Atemu2024-03-02T08:32:40Z2024-03-02T08:32:40Z
<p>Thank you for looking into this again.</p>
<blockquote><p>I'd rather avoid OpenSSL wrappers because adding a C library dependency on openssl will complicate building git-annex in some situations.</p></blockquote>
<p>Would it be possible to make this a build-time option perhaps?</p>
<p>git-annex without SIMD hashing obviously still <em>works</em> fast enough for many purposes as its the status quo but having it would be a greatly appreciated optimisation by many. It'd be great to have the option to enable it wherever possible and simply fall back to non-SIMD where it isn't.</p>
<blockquote><p>Also, whatever library git-annex uses needs to support incremental hashing, otherwise git-annex has to pay a performance penalty of re-reading a file to hash it after download, rather than hashing while downloading.</p></blockquote>
<p>Agreed. Incremental hashing is too important to lose over a general optimisation like this.</p>
comment 13http://git-annex.branchable.com/bugs/git-annex_is_slow_at_reading_file_content/comment_13_0eb12582a3e182b697a07b833cfbe384/joey2024-03-11T14:00:49Z2024-03-11T13:43:19Z
<p><a href="https://hackage.haskell.org/package/botan-low">https://hackage.haskell.org/package/botan-low</a> is another possibility.
There is a significant effort ongoing to build up this library stack in
haskell. It does need the botan C library to be installed separately
unfortunately (I've suggested they embed it).</p>
<p>I have not benchmarked it yet but the docs say it supports hardware
accellerated SHA1, SHA2, SHA3 on x86 and also SHA1, SHA2 on arm64.</p>