add xxHash backend

From https://cyan4973.github.io/xxHash/ , xxHash seems much faster than md5 with comparable quality. There's a Haskell implementation.

RSS Atom

comment 1

I looked at xxHash recently. I can't seem to find benchmarks of it compared with other fast hashes like Blake2.

Comment by joey — Mon Jan 6 19:30:43 2020

comment 2

Let alone blake3, which is 5-6 times as fast as blake2 while still apparently being a cryptographically secure hash.

Comment by joey — Thu Jan 9 20:52:25 2020

comment 3

https://cyan4973.github.io/xxHash/ now includes blake2, and xxh3 is much faster, 28 times as fast.

Would need haskell library, http://hackage.haskell.org/package/xxhash is out of date. It would probably not be hard to make a xxh3 haskell library, but I'm inclined to wait for someone who really wants it.

Debian already has it in libxxhash0.

Interestingly, the ghc RTS uses xxhash and recently updated to xxh3. https://gitlab.haskell.org/ghc/ghc/-/merge_requests/4248 But I don't think that's exposed to haskell code.

Comment by joey — Tue Dec 15 20:23:15 2020

the X prefix conflicts with the eXternal backend namespace

I'm trying to create a external backend for xxHash, but experienced weird behaviors.

If only /bin/git-annex-backend-XXH3 is present in $PATH, and git config annex.backend XXH3 is set, then git annex complains Cannot run git-annex-backend-XH3 -- It is not installed in PATH, which seems like a bug. And if /bin/git-annex-backend-XXH3 is moved to /bin/git-annex-backend-XH3 according to the error message, it will complain Cannot run git-annex-backend-XXH3 -- It is not installed in PATH (this is expected). Finally I have to link the same shell script to both /bin/git-annex-backend-XH3 and /bin/git-annex-backend-XXH3 to make the backend config XXH3 work.

#!/bin/sh

set -e

hashtype="${0##*git-annex-backend-X}"

# could send PROGRESS while doing this, but it's
# hard to implement that in shell
case "$hashtype" in
    BLAKE3_256)
        hashfile() { b3sum --no-names "$1"; } ;;
    BLAKE3_512)
        hashfile() { b3sum --no-names -l 64 "$1"; } ;;
    XXH32|XH32)
        hashfile() { xxhsum -H0 "$1" | cut -d ' ' -f 1; } ;;
    XXH64|XH64)
        hashfile() { xxhsum -H1 "$1" | cut -d ' ' -f 1; } ;;
    XXH128|XH128)
        hashfile() { xxhsum -H2 "$1" | cut -d ' ' -f 1; } ;;
    XXH3|XH3)
        hashfile() { xxhsum -H3 --tag "$1" | awk '{ print $NF }'; } ;;
esac

while read line; do
    set -- $line
    case "$1" in
        GETVERSION)
            echo VERSION 1
        ;;
        CANVERIFY)
            echo CANVERIFY-YES
        ;;
        ISSTABLE)
            echo ISSTABLE-YES
        ;;
        ISCRYPTOGRAPHICALLYSECURE)
            echo ISCRYPTOGRAPHICALLYSECURE-YES
        ;;
        GENKEY)
            contentfile="$2"
            hash=$(hashfile "$contentfile")
            sz=$(wc -c "$contentfile" | cut -d ' ' -f 1)
            if [ -n "$hash" ]; then
                echo "GENKEY-SUCCESS" "$hashtype-s$sz--$hash"
            else
                echo "GENKEY-FAILURE" "calculate hash sum failed"
            fi
        ;;
        VERIFYKEYCONTENT)
            key="$2"
            contentfile="$3"
            hash=$(hashfile "$contentfile")
            khash=$(echo "$key" | sed 's/.*--//')
            if [ "$hash" = "$khash" ]; then
                echo "VERIFYKEYCONTENT-SUCCESS"
            else
                echo "VERIFYKEYCONTENT-FAILURE"
            fi
        ;;
        *)
            echo ERROR protocol error
        ;;
    esac
done

Comment by Arnie97 — Wed Sep 24 12:05:05 2025

comment 5

This is a bug in your program. It is generating a key using the XH3 backend, rather than the XXH3 backend.

[2025-09-24 12:29:41.565937669] (Annex.ExternalAddonProcess) /home/joey/bin/git-annex-backend-XXH3[1] <-- GENKEY .git/annex/othertmp/ingest-bar89415-0
[2025-09-24 12:29:41.568293334] (Annex.ExternalAddonProcess) /home/joey/bin/git-annex-backend-XXH3[1] --> GENKEY-SUCCESS XH3-s30--88ad06d188b880a1

When git-annex later wants to do something that that key, it expects to find a git-annex-backend-XH3 program.

This change will fix it:

- hashtype="${0##*git-annex-backend-X}"
+ hashtype="${0##*git-annex-backend-}"

However, since the hash is named "XXHASH", and this is an external backend, I think the backend name you should really be using is "XXXHASH". This leaves the "XXHASH" backend name free for git-annex to use if it implemented it as a built-in backend.

Once you have the program working, we can add it to the list of external backends.

Comment by joey — Wed Sep 24 16:30:12 2025

comment 6

I am inclined to keep this todo open despite external backend programs existing, because it would be nice to have xxHash in git-annex natively due to its speed.

I found this haskell library which includes xxh3 and which would be easy to add as a git-annex dependency, although it would need to be gated behind a build flag for now: https://hackage.haskell.org/package/xxhash-ffi

(Since that library uses Hashable, it generates an Int for the hash. This seems to limit it to be used on 64 bit platforms. https://github.com/haskell-haskey/xxhash-ffi/issues/6 The lower-level Data.Digest.XXHash.FFI.C uses CULLong so will work on 32 bit.)

Comment by joey — Wed Sep 24 16:39:02 2025

Add a comment