tips/using Amazon S3git-annexhttp://git-annex.branchable.com/tips/using_Amazon_S3/git-annexikiwiki2017-09-29T17:46:42ZANNEX_S3 vs AWS for keyshttp://git-annex.branchable.com/tips/using_Amazon_S3/comment_1_666a26f95024760c99c627eed37b1966/Matt2013-11-27T22:47:37Z2012-05-29T12:24:25Z
The instructions state ANNEX_S3_ACCESS_KEY_ID and ANNEX_SECRET_ACCESS_KEY but git-annex cannot connect with those constants. git-annex tells me to set both "AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY" instead, which works. This is with Xubuntu 12.04.
comment 2http://git-annex.branchable.com/tips/using_Amazon_S3/comment_2_f5a0883be7dbb421b584c6dc0165f1ef/joeyh.name2013-11-27T22:47:37Z2012-05-29T19:10:42Z
Thanks, I've fixed that. (You could have too.. this is a wiki <img src="http://git-annex.branchable.com/smileys/smile4.png" alt=";)" />
Altering AWS credentialshttp://git-annex.branchable.com/tips/using_Amazon_S3/comment_3_32acba030c2ad252e2f7027075e4303e/annexuser2014-04-15T21:59:43Z2014-04-15T21:59:43Z
If I revoke old AWS credentials and create new ones, how would I inform git-annex of the change to <code>AWS_ACCESS_KEY_ID</code> and <code>AWS_SECRET_ACCESS_KEY</code>?
comment 4http://git-annex.branchable.com/tips/using_Amazon_S3/comment_4_92df5a9f923beafba55a1c455728112e/joeyh.name2014-04-17T19:44:55Z2014-04-17T19:44:55Z
<p>You can use <code>git annex enableremote</code> to change an existing remote's configuration. So this should work:</p>
<pre><code># export AWS_ACCESS_KEY_ID="newRANDOMGOBBLDEYGOOK"
# export AWS_SECRET_ACCESS_KEY="news3kr1t"
# git annex enableremote cloud
</code></pre>
What if you do not want to encrypt?http://git-annex.branchable.com/tips/using_Amazon_S3/comment_5_6b288e06010bedbb71e0afed6b427704/Jack William2014-09-07T18:35:19Z2014-09-07T18:35:19Z
Once use case for GIT with Amazon S3 is to maintain a web site on S3 you can easily update from a local machine. In that case you would not want to encrypt. Is encryption optional? This isn't clear from the instructions.
comment 6http://git-annex.branchable.com/tips/using_Amazon_S3/comment_6_3b6c74b40ea666389ac04b52ab0819a0/Schnouki2014-09-09T12:48:58Z2014-09-09T12:48:59Z
<p>Jack, if you don't want to use encryption you can use <code>encryption=none</code> as documented <a href="http://git-annex.branchable.com/special_remotes/S3/">here</a>.</p>
<p>I'm not sure exactly what you're trying to do, but please note that you files won't be easily available on S3: they will be named as git-annex keys, with long and unreadable names such as "SHA256E-s6311--c7533fdd259d872793b7298cbb56a1912e80c52a845661b0b9ff391c65ee2abc.html" instead of "index.html".</p>
How to publish your files to the publichttp://git-annex.branchable.com/tips/using_Amazon_S3/comment_7_cf6755d88463878f2ea6e4c300899027/Giovanni2014-09-13T15:29:18Z2014-09-13T15:29:18Z
<p>I don't know if this is what Jack wanted, but you can upload your files to S3 and let them be accessible through a public URL.</p>
<p>First, go to (or create) the bucket you will use at <a href="https://console.aws.amazon.com/s3/">S3</a> and add a public get policy to it:</p>
<pre><code> {
"Version": "2008-10-17",
"Statement": [
{
"Sid": "AllowPublicRead",
"Effect": "Allow",
"Principal": {
"AWS": "*"
},
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::BUCKETNAME/*"
}
]
}
</code></pre>
<p>Then set up your special remote with the options <code>encryption=none</code>, <code>bucket='BUCKETNAME'</code> <code>chunk=0</code> (and any others you want).</p>
<p>Your files will be accessible through <code>http://BUCKETNAME.s3-website-LOCATION.amazonaws.com/KEY</code> where location is the one specified through the options <code>datacenter</code> and KEY is the SHA-SOMETHING hash of the file, created by git annex and accessible if you run <code>git annex lookupkey FILEPATH</code>.</p>
<p>This way you can share a link to each file you have at your S3 remote.</p>
comment 8http://git-annex.branchable.com/tips/using_Amazon_S3/comment_8_4f9c2f6627f8ed3423bcc8b7bf2f76cb/Lemao2015-01-07T13:54:23Z2015-01-07T13:54:23Z
<p>I use github as my central git repository and I would like to use S3 to store large files with annex. Since the s3 remote in .git/config is not stored in github, how do I make sure I reconnect to the same s3 bucket in case I delete my local clone? Reinitializing the remote will create a completely new bucket.</p>
<p>I would also be a good idea to centralize git-annex folders inside a single bucket so I keep the global namespace under control and can narrow down the permissioning.</p>
comment 9http://git-annex.branchable.com/tips/using_Amazon_S3/comment_9_47e4ea77d0262d332d86a06d7aaeddd8/joeyh.name2015-01-07T17:25:43Z2015-01-07T17:25:43Z
<p>Lemao, make sure you have pushed your git-annex branch to your central git repository.</p>
<p>When you clone that repo elsewhere, you can add the S3 remote by running <code>git annex enableremote cloud</code> (replace "cloud" with whatever name you originally picked when you used <code>git annex initremote</code> to set up the S3 remote in the first place.</p>
<p>git-annex stores the necessary configuration of the S3 remote on the git-annex branch.</p>
comment 10http://git-annex.branchable.com/tips/using_Amazon_S3/comment_10_43f3f12a83e3ace1674eae395b865409/Lemao2015-01-07T23:53:32Z2015-01-07T23:53:32Z
My bad. I didn't realize git annex creates a new branch. After I pushed it, the clone was able to enableremote and I was able to get the file.
comment 11http://git-annex.branchable.com/tips/using_Amazon_S3/comment_11_70b4e62aabb93d078811cbdcd3cced20/James2015-01-22T22:16:46Z2015-01-22T22:16:46Z
<p>Even after enableremote I can't get from s3.</p>
<pre><code>get Docs.zip (from s3...)
Unable to access these remotes: s3
Try making some of these repositories available:
02b4e373-02b1-42b9-8285-c353c047895c -- my laptop
f287b4f7-4e5a-4734-bd6b-62b1f4498838 -- [s3]
(Note that these git remotes have annex-ignore set: origin)
failed
git-annex: get: 1 failed
</code></pre>
<p>This is after all branches are pushed from my original repo. Any suggestions?</p>
RE: last commenthttp://git-annex.branchable.com/tips/using_Amazon_S3/comment_12_d00472bf4ac5f7b2212073c081ba04c2/James2015-01-23T01:02:18Z2015-01-23T01:02:18Z
<p>RE: my last comment</p>
<p>The reason I couldn't get it to work is because I didn't have proper read access to the bucket. My bad for not checking first but it would be great it there was a clearer error message from git-annex and/or a way to get more detailed information on the s3 extension (-d doesn't do much).</p>
<p>Regardless git-annex is pretty cool, thanks to all the maintainers for their hard work.</p>
enable s3 remote on clonehttp://git-annex.branchable.com/tips/using_Amazon_S3/comment_13_30bdbd217fd2b603984cf7d3a3dce266/james2015-07-03T14:46:49Z2015-07-03T14:46:49Z
<p>Hi,
I am trying to enable access to my s3 area from a clone. I am running into this issue:</p>
<pre><code>$ git annex enableremote mys3
enableremote mys3 (encryption update) (hybrid cipher with gpg key EA1CF14BD8467AFB) (gpg) gpg: decryption failed: secret key not available
git-annex: user error (gpg ["--batch","--no-tty","--use-agent","--quiet","--trust-model","always","--decrypt"] exited 2)
failed
git-annex: enableremote: 1 failed
</code></pre>
<p>My gpg key is available :</p>
<pre><code>$ gpg -K EA1CF14BD8467AFB
sec 4096R/D8467AFB 2010-10-25
uid James Richardson (email) <james@jamestechnotes.com>
uid James Richardson <james.richardson.jr@gmail.com>
uid [ revoked] James Richardson (list) <jr@jamesr.biz>
uid [ revoked] James Richardson (James Richardson) <james@jamesr.biz>
ssb 4096R/F90CF7F0 2010-10-25
ssb 4096R/005D609B 2010-10-26
</code></pre>
<p>I would expect this to pop up a dialog asking me for my passphrase, as it will when I run the gpg command from a term.</p>
<p>Any ideas?</p>
comment 14http://git-annex.branchable.com/tips/using_Amazon_S3/comment_14_5bb12e6fa51da00ecf48f2dee1bfb050/joey2015-07-06T18:41:07Z2015-07-06T17:42:28Z
<p>@james, since your keyring apparenty contains your secret key, the problem
may be in the configuration of your gpg agent or pinentry program. If the agent
is unable to use pinentry for some reason, gpg will complain that the
secret key is unavailable since it is unable to get the passphrase to
unlock it.</p>
<p>I had a similar problem with gpg2 the other day:
<a href="http://bugs.debian.org/791379">http://bugs.debian.org/791379</a></p>
The initremote command appears to hang due to low entropyhttp://git-annex.branchable.com/tips/using_Amazon_S3/comment_15_d8cc20706debc17f4f738d2019577dea/NathanCollins2017-09-10T02:32:26Z2017-09-10T02:32:26Z
<p>For me, the <code>git annex initremote amazon-s3 encryption=shared embedcreds=yes</code> [1] command hung for several minutes after printing</p>
<pre><code>initremote amazon-s3 (encryption setup)
</code></pre>
<p>Turns out the problem was that I was low on entropy. Figured this out by running</p>
<pre><code>gpg --gen-random 2
</code></pre>
<p>per <a href="https://github.com/DanielDent/git-annex-remote-rclone/issues/6#issuecomment-231347642">this bug comment</a>. According to <a href="https://delightlylinux.wordpress.com/2015/07/01/is-gpg-hanging-when-generating-a-key/">this blog post</a> a solution is to</p>
<pre><code>sudo aptitude install rng-tools
sudo rngd -r /dev/urandom
</code></pre>
<p>The <code>git annex initremote</code> command had finished by the time I found that solution, but I verified that it made <code>gpg --gen-random 2</code> work.</p>
<p>System: Ubuntu 16.04 with Git Annex 5.20151208-1build1 installed via package manager.</p>
<p>[1] I'm using AWS credentials that are restricted to a specific bucket, so I'm not worried about the conjunction <code>encryption=shared</code> and <code>embedcreds=yes</code>.</p>
comment 16http://git-annex.branchable.com/tips/using_Amazon_S3/comment_16_bb2cb128d9c6a8e9176604cef1f6fc91/joey2017-09-29T17:46:42Z2017-09-29T17:45:59Z
<p>You can also use the --fast option to make git-annex use less entropy
when generating the encryption key. That's a little less secure, but
probably still secure enough.</p>