Please describe the problem.
Uploading a 21GB file to an S3 special remote fails. It will generally fail somewhere at about 3-15%. I am using the new chunking feature, with chunks set to 25MiB.
What steps will reproduce the problem?
$ git annex copy my-big-file.tar.bz --to s3
copy my-big-file.tar.bz (gpg) (checking s3...) (to s3...)
13% 863.8KB/s 6h0m
ErrorClosed
failed
git-annex: copy: 1 failed
What version of git-annex are you using? On what operating system?
Running on Arch Linux.
git-annex version: 5.20140818-g10bf03a
build flags: Assistant Webapp Webapp-secure Pairing Testsuite S3 WebDAV Inotify DBus DesktopNotify XMPP DNS Feeds Quvi TDFA CryptoHash
key/value backends: SHA256E SHA1E SHA512E SHA224E SHA384E SKEIN256E SKEIN512E SHA256 SHA1 SHA512 SHA224 SHA384 SKEIN256 SKEIN512 WORM URL
remote types: git gcrypt S3 bup directory rsync web webdav tahoe glacier ddar hook external
local repository version: 5
supported repository version: 5
upgrade supported from repository versions: 0 1 2 4
Please provide any additional information below.
If I fire up the web app and open the log, the end looks like this:
...
3% 857.3KB/s 6h46m
3% 857.3KB/s 6h46m
3% 857.3KB/s 6h46m
3% 857.4KB/s 6h46m
3% 857.4KB/s 6h46m
3% 857.5KB/s 6h46m
3% 857.5KB/s 6h46m
3% 857.6KB/s 6h46m
3% 857.6KB/s 6h46m
3% 857.6KB/s 6h46m
3% 857.7KB/s 6h46m
3% 857.7KB/s 6h46m
3% 857.8KB/s 6h46m
3% 857.8KB/s 6h46m
3% 857.8KB/s 6h46m
3% 857.9KB/s 6h46m
3% 857.9KB/s 6h46m
3% 858.0KB/s 6h46m
3% 858.0KB/s 6h46m
3% 858.1KB/s 6h46m
3% 858.1KB/s 6h45m
3% 858.1KB/s 6h45mmux_client_request_session: read from master failed: Broken pipe
This is using the old hS3 library. So, each chunk is sent using a new http connection. It seems that the connection must be being closed by S3 part way through the upload of a chunk.
It may be that the new aws library somehow avoids this problem. So, a git-annex built with the
s3-awsbranch merged in may help with this bug. OTOH, that new branch makes a single http connection be reused for all the chunks in a file, so it might also make things worse.If you're using the new chunking system, git-annex should support resuming the upload to S3. Next time you try to send the file, it should find the chunks that were successfully sent, and resume at the chunk where it failed.
Supporting this even for encrypted uploads was a major benefit of the new chunking system, so I hope it works...?
s3-awsbranch? How do I determine if my version of git-annex was built with the new or old chunking?git show git-annex:remote.log, find the line for the UUID of the remote, and see if it has chunk= (new chunking) or chunksize= (old chunking).I am experiencing similar behavior on Ubuntu Trusty (x86_64) using a prebuilt Linux release:
Copying files to S3 consistently fails both from the command line and via the assistant:
Two files (out of several hundred) have succeeded.
Any ideas?
I need to know if the S3 remote is configured to use the new style chunking feature, and what size chunks it is configured to use. I have already explained how to check that in this thread.
I also need to know if retrying the upload after it fails lets it resume where it left off.
It is running the new style chunking (chunk=1MiB).
It does not appear to resume when it tries again. If I try copying a file to the remote from the command line, it always starts at 0% and dies at some point before 100% even if it has tried to copy that file before.
When it resumes, it will start at 0% but jump forward to the resume point pretty quickly, after verifying which chunks have already been sent. If any full chunk gets transferred, I'd expect it to resume. This may not be very obvious it's happening for smaller files.
I have been running
git annex testremoteagainst S3 special remotes today, and have not managed to reproduce this problem (using either the old S3 or the new AWS libraries). It could be anything, including a problem with your network or the network between you and the S3 endpoint. Have you tried using a different S3 region?