Please describe the problem.
When copying to a bup remote with -J many copies "fail":
copy samba-2.2.3a/packaging/LSB/lsb-samba.spec (to bup...)
user error (bup ["split","-r","/tmp/silesia_bup","-q","-n","SHA256E-s2260--1e95bcc60c9b332608774459570b80ab9576b5e971013a30ff6da6d93ebcfd9d.spec"] exited 1)
They actually seem to complete in some sense though as trying to copy them again will print ok instantly.
What steps will reproduce the problem?
git annex copy --to bup -J2
What version of git-annex are you using? On what operating system?
git-annex version: 10.20220624
build flags: Assistant Webapp Pairing Inotify DBus DesktopNotify TorrentParser MagicMime Feeds Testsuite S3 WebDAV
dependency versions: aws-0.22 bloomfilter-2.0.1.0 cryptonite-0.29 DAV-1.3.4 feed-1.3.2.1 ghc-9.0.2 http-client-0.7.11 persistent-sqlite-2.13.1.0 torrent-10000.1.1 uuid-1.3.15 yesod-1.6.2
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL X*
remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs httpalso borg hook external
operating system: linux x86_64
supported repository versions: 8 9 10
upgrade supported from repository versions: 0 1 2 3 4 5 6 7 8 9 10
local repository version: 8
Please provide any additional information below.
# If you can, paste a complete transcript of the problem occurring here.
# If the problem is with the git-annex assistant, paste in .git/annex/daemon.log
# End of transcript or log.
Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)
It's amazing and I use it every day 
This seems most likely to be a concurrency issue in bup itself, or in how it uses git.
However, I have not been able to reproduce it so far. I tried making 8 different files each 100 mb, and was able to copy them all to a bup repository concurrently with no failures.
That was with bup 0.32. What is your bup version?
It would be good to know what error message bup outputs. Unfortunately, since
bup split -qis noisy on stderr, git-annex sinks it to /dev/null and so you don't see it.I've changed git-annex to display the stderr when
bup splitfails. So if you can get an updated build of git-annex, and reproduce this problem, we'll be able to see if this is a bug in bup, or what.Thanks for looking into this!
You need more than that; 8 Files might randomly work without error.
Perhaps it only really manifests itself on smaller files? That's mainly what I've been copying.
Here's an example output of git-annex 6d762fafae45cb1f6a6106ac25f347631d6d0ca8
git-annex copy --not --in bup --to bup -J2 --all:(Ran this on aarch64-darwin because that's all I have access to ATM but x86_64-linux will probably have the same errors.)
Given the error, it indeed looks like a concurrency issue in
bupitself. Version is 0.32 which is the newest.Looks like bup-midx is at fault, and that makes sense; it converts git pack files into "midx" files, so is reading and deleting pack files. So it seems likely that two of those running at the same time would fail like this.
I've posted to bup's mailing list about this: https://groups.google.com/g/bup-list/c/SJg6rQ2XWy8
git-annex may need to work around this by only running one bup-split at a time.
Got confirmation that bup is generally not concurrency safe.
I've made git-annex limit the number of bup-split it runs to 1.
It may be that this will also need to be done with bup-join, but I think probably not since it probably does not write to the repo, and a bup-split is unlikely to get in its way.
Thank you!
That's sad to hear as it means bup will perform even worse where it wasn't performing well to begin with but unsafe is unsafe.
I'd consider this issue solved as bup performance problems are tracked in https://git-annex.branchable.com/bugs/Copying_many_files_to_bup_remotes_is_very_slow/ already.