forum/Several concurrent operationsgit-annexhttp://git-annex.branchable.com/forum/Several_concurrent_operations/git-annexikiwiki2019-01-21T15:42:51Zcomment 1http://git-annex.branchable.com/forum/Several_concurrent_operations/comment_1_885edabc67687e727ab1a111d7d10d1f/joey2018-08-15T15:43:59Z2018-08-15T15:43:00Z
git-annex is concurrency safe
comment 2http://git-annex.branchable.com/forum/Several_concurrent_operations/comment_2_581e53306e47ddb826c030333f9ae9c2/liori2018-08-17T21:58:03Z2018-08-17T21:58:03Z
So, indeed it works. However, syncing a large archive (2TB, ~1M objects) over several USB HDDs at the same time ended up a bad idea: inevitably at some point every process tried syncing with the same, slowest, repository, and lack of good I/O concurrency on a USB-connected HDD killed all performance. I'm guessing that introducing some targetted locking could probably help, or even maybe just randomizing the order in which repositories are pulled from/pushed to.
comment 3http://git-annex.branchable.com/forum/Several_concurrent_operations/comment_3_fe49537fa9e0a284705543b010e7aea9/joey2019-01-21T15:42:51Z2018-10-08T16:42:29Z
<p>If you use the --jobs option, git-annex will actually balance load amoung
multiple remotes that have the same cost. Which will prevent everything
bottlenecking on a single repository.</p>
<p>Alternatively, configure the cost of the slow repository to be higher than
the other drives, and then things like <code>git annex get</code> will avoid using the
slow one if a faster one is available.</p>