Please describe the problem.

I'm trying to copy a directory full of photos (~9GB) to an S3 special remote. It seems to fail when running multiple jobs. The machine I am using has two cores (4 logical due to hyperthreading) and I initially used -J8 and that failed quickly. I haven't retried that yet to see if it always fails quickly but I immediately retried with -J4 and that seemed to run okay for a little over an hour and a half before failing again. This is the second large directory I've tried to copy to this remote and had a similar problem the last time (many retries seemed to allow me to get through).

What steps will reproduce the problem?

What version of git-annex are you using? On what operating system?

git-annex was installed via homebrew on macos 10.14.5

15:23 $ git annex version
git-annex version: 7.20190322
build flags: Assistant Webapp Pairing S3(multipartupload)(storageclasses) WebDAV FsEvents TorrentParser MagicMime Feeds Testsuite
dependency versions: aws-0.19 bloomfilter- cryptonite-0.25 DAV-1.3.3 feed- ghc-8.2.2 http-client-0.5.14 persistent-sqlite-2.6.4 torrent-10000.1.1 uuid-1.3.13 yesod-1.4.5
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLA
remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar hook external
operating system: darwin x86_64
supported repository versions: 5 7
upgrade supported from repository versions: 0 1 2 3 4 5 6

Please provide any additional information below.

22:56 $ git annex copy -J8 photos/ --to s3
copy photos/2001/01/IMAG0003.JPG (checking s3...) (to s3...) ok
copy photos/2001/01/IMAG0006.JPG (checking s3...) (to s3...)
88%   319.84 KiB       68 KiB/s 0s
copy photos/2001/01/IMAG0001-1.JPG (checking s3...) (to s3...)
63%   255.88 KiB       25 KiB/s 6s
copy photos/2001/01/IMAG0002.JPG (checking s3...) (to s3...)
63%   223.89 KiB       17 KiB/s 7s
copy photos/2001/01/IMAG0004.JPG (checking s3...) (to s3...)
copy photos/2001/01/IMAG0005.JPG (checking s3...) (to s3...)
80%   287.86 KiB      159 KiB/s 0s
copy photos/2001/01/IMAG0001.JPG (checking s3...) (to s3...)
74%   255.88 KiB       49 KiB/s 1s
copy photos/2001/01/IMAG0002-1.JPG (checking s3...) (to s3...)
  ParseError {errorContexts = [], errorMessage = "Failed reading: takeWhile1", errorPosition = 4:2 (285)}
copy photos/2001/01/IMAG0007.JPG (checking s3...) (to s3...)
10%   31.98 KiB       102 KiB/s 2s
git-annex(21112,0x70000dd5b000) malloc: Incorrect checksum for freed object 0x7fe0b3e02bf8: probably modified after being freed.
Corrupt value: 0x6574636f2f6e6f69
git-annex(21112,0x70000dd5b000) malloc: *** set a breakpoint in malloc_error_break to debug
error: git-annex died of signal 6

Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)

I'm a new git-annex user and besides these occasional issues with concurrent with git-annex-copy I found it really easy to get going with git-annex and to manage my imported files. In particular I really appreciate the extensive documentation!