Future proofing / disaster recovery with an encrypted special remote

I am interested in using git annex to manage encrypted backups to Amazon S3/Glacier. So git annex will be used with the main file directory in direct mode and an encrypted S3 or Glacier remote set up in archive mode and then git annex add . and git annex sync will be run periodically. The intent is for this set up to be a backup for catastrophic failure, so I want to make sure I take care of future-proofing and disaster recovery properly. So my basic question is what would I need to have backed up and what would I have to do if the computer with the main repository died. I try to break that out into more specific questions below.

S3/Glacier remotes store the contents of .git/annex/objects in encrypted form with hashes for file names and nothing else (other than a uuid). The hashes do not match the keys in the main repo. Are they the same keys encrypted? Is there a way to look up the S3 file name corresponding to a file in the repo?
For shared encryption, I see the cipher text in remote.log in the git-annex branch. Assuming I didn't have access to git annex, what would I need to do to convert that cipher text into a form that I could use with gpg to decrypt files?
Same question but for hybrid encryption rather than shared. I assume the answer is similar but I need to decrypt the cipher first with my gpg key? How do I do that?
Assuming I did have access to git annex, what would I need to create a new repo on a new computer with access to all of the files in the S3/Glacier bucket? I think I would need my Amazon credentials (possibly already embedded in the git repo), my gpg key if using hybrid or public key encryption, and the .git folder as it was the last time files were pushed to the S3/Glacier remote (which would have the necessary decryption information for shared encryption). Is that right? I guess mainly I am checking that the remote does not store any metadata about the repo, so for git annex to be able to pull files back out I would need a backup of the .git directory and that back up would need to be up to date (can't just copy remote.log and have git annex work out the rest from the remote's contents). So for a full backup, my script would need to tar the .git directory, encrypt it, and push it to S3/Glacier separately after git annex does a sync. Then I could recover everything as long as I had a secure backup of my Amazon credentials and my encryption key(s).

RSS Atom

comment 1

That's what's "special" about special remotes vs regular git remotes: They only store the content of annexed files and not the git repository. Back up the git repository separately (and your gpg key if it's being used, and the credentials if you didn't use embedcreds=yes)

To use your backup, you can make a clone of the backed up git repository and use git annex enableremote to enable it to use the special remote.

See encryption for details of how the encryption is implemented. I've seen people follow that and manually use the data from the git repo to decrypt files, but I don't have a pointer to an example at the moment.

Comment by joey — Fri Dec 11 14:42:32 2015

Remove comment

comment 2

Thanks for the response.

With pubkey encryption, I am able to decrypt the remote's files normally using just gpg --decrypt with the public key used to encrypt them in my keyring. One thing I don't understand about pubkey encryption: what is in the cipher= entry in remote.log after the 256 bytes representing the HMAC cipher? The gpg key pair is used for encryption, so there is no encryption cipher to put in remote.log after the HMAC cipher, I would have thought.

I understand the basic encryption set up, but I don't know how to use gpg to work with a raw cipher or raw encrypted text. For example with pubkey encryption, my interpretation is that the cipher= entry in remote.log is encrypted with the public key, but I can't just pass teh entry to gpg --decrypt because gpg expects the encrypted input to be formatted in a way that specifies which key to use to decrypt. Otherwise it says gpg: no valid OpenPGP data found. Similarly for shared encryption, I don't see how to pass the second half of the remote.log cipher= entry to gpg to decrypt the remote's files.

I am also having trouble generating the special remote's keys. I created a directory remote with shared encryption (so that the cipher= entry in remote.log would not be encrypted). Then I added one file called tmp.txt which was stored in the annex as SHA256E-s9--9c73fdec185d79405f58fc8b4e0ac22fa5ed2de7b7611a61b37606c905509650.txt I did git checkout git-annex -- remote.log and then tried the following:

    echo -n "SHA256E-s9--9c73fdec185d79405f58fc8b4e0ac22fa5ed2de7b7611a61b37606c905509650.txt" | openssl dgst -sha1 -hmac $(echo -n $(grep -oP 'cipher\=.*? ' remote.log | sed 's/cipher=//') | base64 -d | xxd -p -l 256 -c 256) -macopt hexkey:string`

but the output does not match the GPGHMACSHA1 file name in the remote, and I don't understand why. I tried other variations as well (dropping the .txt or the SHA256E-s9--` prefix) but they did not work either.

Comment by wsha.code+ga — Sat Dec 12 10:42:46 2015

Remove comment

comment 3

Small update: I have gotten the HMAC and decryption steps to work for shared encryption. I didn't figure out what was wrong with my HMAC command above yet, but I was able to reproduce the special remote's keys in Python using the hmac and hashlib libraries. I was trying to hash the correct string in the comment above (the full key with the SHA... prefix and the file extension suffix). For decryption, I used gpg on the encrypted file in the special remote and passed it the cipher from remote.log with the first 256 bytes removed as the passphrase (in the format returned by base64.b64decode() in Python).

I still need to figure out how to decrypt the ciphers for pubkey and hybrid.

I will try to put together a tip with the steps needed to reproduce special remote keys and to decrypt special remote files using only command line tools after that (assuming I can translate the Python steps back to command line tools; otherwise part of the steps will be in Python).

Comment by wsha.code+ga — Mon Dec 14 12:03:22 2015

Remove comment

comment 4

Decrypting the cipher for hybrid and pubkey remotes is actually pretty straightforward: echo -n <cipher_entry> | base64 -d | gpg --decrypt. I think with that I have enough to put together a short summary of decrypting by hand the contents of hybrid, pubkey, and shared special remotes.

Comment by wsha.code+ga — Wed Dec 16 11:21:48 2015

Remove comment

comment 5

When using pubkey, the second 256 bytes of ciphertext are currently not used for anything.

For hybrid and shared, the 256 bytes of ciphertext are used as a symmetric cipher. So the gpg option to use for both encrypting and decrypting is --symmetric. gpg then prompts for a passphrase, and the ciphertext is what's used.

Comment by joey — Sat Dec 19 17:59:38 2015

Remove comment

comment 6

joey, thanks for answering my questions. I have put together a bash script to check my understanding of how the different encryption methods work. The script is below. With it, I am able to generate the encrypted keys for items for hybrid and shared encrypted directory remotes and to decrypt items for hybrid, shared, and pubkey encrypted directory remotes (and presumably other special remotes but I tested with directory remotes since those were simplest). The only thing I have not been able to get to work is the generation of the encrypted item keys for pubkey remotes. Do you see what I am doing wrong for that case? Once I figure that out, I can make a tip entry on the wiki with the script.

#!/usr/bin/env bash

# args: file_path remote decrypt decrypt_path

# For shared, pubkey, and hybrid encryption:
# 1. Take in symlink and output encrypted key
# 2. Take in path to encrypted file and print decrypted file to stdout
#
# args: -r remote [-k symlink] [-d encrypted_file_path]

usage() {
    echo "Usage: ga_decrypt.sh -r REMOTE [-k SYMLINK] [-d FILE]"
    echo ""
    echo "    Either lookups up key on REMOTE for annex file linked with SYMLINK"
    echo "    or decrypts FILE encrypted for REMOTE."
    echo ""
    echo "    -r: REMOTE is special remote to use"
    echo "    -k: SYMLINK is symlink in annex to print encrypted special remote key for"
    echo "    -d: FILE is path to special remote file to decrypt to STDOUT"
    echo ""
    echo "NOTES: "
    echo "    * Run in an indirect git annex repo."
    echo "    * Must specify -k or -d."
    echo "    * -k prints the key including the leading directory names used for a "
    echo "       directory remote (even if REMOTE is not a directory remote)"
    echo "    * -d works on a locally accessible file. It does not fetch a remote file"
    echo "    * Must have gpg and openssl"
}

decrypt_cipher() {
    cipher="$1"
    echo "$(echo -n "$cipher" | base64 -d | gpg --decrypt --quiet)"
}

lookup_key() {
    encryption="$1"
    cipher="$2"
    symlink="$3"

    if [ "$encryption" == "hybrid" ] || [ "$encryption" == "pubkey" ]; then
        cipher="$(decrypt_cipher "$cipher")"
    fi

    # Pull out MAC cipher from beginning of cipher
    if [ "$encryption" = "hybrid" ] ; then
        cipher="$(echo -n "$cipher" | head  -c 256 )"
    elif [ "$encryption" = "shared" ] ; then
        cipher="$(echo -n "$cipher" | base64 -d | tr -d '\n' | head  -c 256 )"
    elif [ "$encryption" = "pubkey" ] ; then
        # ???
        : # Use entire base64 decrypted cipher
        # cipher="$(echo -n "$cipher" | head  -c 256 )"
        # cipher="$(echo -n "$cipher" | base64 -d | tr -d '\n' | head  -c 256 )"
        # cipher="$(echo -n "$cipher" | base64 -d | tr -d '\n' )"
    fi

    annex_key="$(basename "$(readlink "$symlink")")"
    hash="$(echo -n "$annex_key" | openssl dgst -sha1 -hmac "$cipher" | sed 's/(stdin)= //')"
    key="GPGHMACSHA1--$hash"
    checksum="$(echo -n $key | md5sum)"
    echo "${checksum:0:3}/${checksum:3:3}/$key"
}

decrypt_file() {
    encryption="$1"
    cipher="$2"
    file_path="$3"

    if [ "$encryption" = "pubkey" ] ; then
        gpg --quiet --decrypt "${file_path}"
    else
        if [ "$encryption" = "hybrid" ] ; then
            cipher="$(decrypt_cipher "$cipher" | tail -c +257)"
        elif [ "$encryption" = "shared" ] ; then
            cipher="$(echo -n "$cipher" | base64 -d | tr -d '\n' | tail  -c +257 )"
        fi
        gpg --quiet --batch --passphrase "$cipher" --output - "${file_path}"
    fi
}

main() {
    OPTIND=1

    mode=""
    remote=""

    while getopts "r:k:d:" opt; do
        case "$opt" in
            r)  remote="$OPTARG"
                ;;
            k)  if [ -z "$mode" ] ; then
                    mode="lookup key"
                else
                    usage
                    exit 2
                fi
                symlink="$OPTARG"
                ;;
            d)  if [ -z "$mode" ] ; then
                    mode="decrypt file"
                else
                    usage
                    exit 2
                fi
                file_path="$OPTARG"
                ;;
        esac
    done

    if [ -z "$mode" ] || [ -z "$remote" ] ; then
        usage
        exit 2
    fi

    shift $((OPTIND-1))

    # Pull out config for desired remote name
    remote_config="$(git show git-annex:remote.log | grep 'name='"$remote ")"

    # Get encryption type and cipher from config
    encryption="$(echo "$remote_config" | grep -oP 'encryption\=.*? ' | tr -d ' \n' | sed 's/encryption=//')"
    cipher="$(echo "$remote_config" | grep -oP 'cipher\=.*? ' | tr -d ' \n' | sed 's/cipher=//')"

    if [ "$mode" = "lookup key" ] ; then
        lookup_key "$encryption" "$cipher" "$symlink"
    elif [ "$mode" = "decrypt file" ] ; then
        decrypt_file "$encryption" "$cipher" "${file_path}"
    fi
}

main "$@"

Comment by wsha.code+ga — Sat Dec 26 12:31:13 2015

Remove comment

comment 7

I think it's great you're working on this, and future proofing is probably a good place to put the result.

I did some digging, and the cipher used for pubkey includes a trailing new line. The trailing newline seems to be lost in your script, and adding it makes it work.

cipher="$cipher
"

I think the actual problem in the script is in this line:

cipher="$(decrypt_cipher "$cipher")"

I verified that decrypt_cipher outputs the trailing newline, but in capturing its output, the shell chomps it. Can't remember ATM if there's a way to avoid that in shell scripting.

Arguably git-annex should not be treating that newline as part of the cipher text, although it's probably too late to change that, certainly for existing repositories..

Comment by joey — Fri Jan 1 20:42:34 2016

Remove comment

comment 8

Thanks for spending the time to check the script. It would have taken me quite a while to catch that issue. I was also a little bit surprised that the hybrid encryption mode uses the base64 encoded cipher string while the shared encryption mode uses the cipher after decoding from base64.

I can now reproduce the HMAC'ed keys and decrypt the files all the supported forms of encryption using openssl and gpg which makes me feel comfortable using them for long term storage.

Comment by wsha.code+ga — Fri Jan 8 06:56:02 2016

Remove comment

comment 9

I'm trying to add support for the sharedpubkey encryption scheme, and while decrypting the files is straight-forward, I cannot get the "lookup_key" function to work with this scheme. I tried the following:

# Pull out MAC cipher from beginning of cipher
if [ "$encryption" = "hybrid" ] ; then
    cipher="$(echo -n "$cipher" | head  -c 256 )"
elif [ "$encryption" = "shared" ] ; then
    cipher="$(echo -n "$cipher" | base64 -d | tr -d '\n' | head  -c 256 )"
elif [ "$encryption" = "pubkey" ] ; then
    # pubkey cipher includes a trailing newline which was stripped in
    # decrypt_cipher process substitution step above
    IFS= read -rd '' cipher < <( printf "$cipher\n" )
elif [ "$encryption" = "sharedpubkey" ] ; then
    cipher="$(echo -n "$cipher" | base64 -d | tr -d '\n' | head -c 256)"
fi

as well as many other variations (full cipher, no base 64 etc...) without success. The cipher is said to be unencrypted, so I guess it is base-64 encoded only. Would it be possible that an extra character has been added to the cipher like in the "pubkey" encryption scheme ?

My tests are done in a directory special remote, sharedpubkey encryption, no chunks, a single test file in the repository.

Comment by oliv5 — Sun Jul 15 23:59:42 2018

Remove comment

comment 10

@oliv5 sharedpubkey's cipher has the same newline problem as pubkey does, as discussed above. Unlike pubkey, it has to be base64-decoded first, and then the extra newline appended to that.

                # Pull out MAC cipher from beginning of cipher
                if [ "$encryption" = "hybrid" ] ; then
                        cipher="$(echo -n "$cipher" | head  -c 256 )"
                elif [ "$encryption" = "shared" ] ; then
                        cipher="$(echo -n "$cipher" | base64 -d | head  -c 256 )"
                elif [ "$encryption" = "pubkey" ] ; then
                        cipher="$cipher
"
                elif [ "$encryption" = "sharedpubkey" ] ; then
                        cipher="$(echo -n "$cipher" | base64 -d)"
            cipher="$cipher
"
        fi

Note that base64 -d does emit the newline (verified with hexdump); again the shell is shooting you in the foot by eliminating it.

BTW, a very simple code hack that makes it easy to dump out the cipher git-annex is using:

diff --git a/Remote/Helper/Encryptable.hs b/Remote/Helper/Encryptable.hs
index 97e55a415..c4d252912 100644
--- a/Remote/Helper/Encryptable.hs
+++ b/Remote/Helper/Encryptable.hs
@@ -192,7 +192,7 @@ describeCipher :: StorableCipher -> String
 describeCipher c = case c of
    (SharedCipher _) -> "encryption key stored in git repository"
    (EncryptedCipher _ _ ks) -> showkeys ks
-   (SharedPubKeyCipher _ ks) -> showkeys ks
+   (SharedPubKeyCipher c ks) -> show c ++ " " ++ showkeys ks
   where
    showkeys (KeyIds { keyIds = ks }) = "to gpg keys: " ++ unwords ks

Then git-annex info remote will display it. Obviously, this patch is insecure.

Comment by joey — Thu Jul 19 16:16:14 2018

Remove comment

Add a comment