devblog/day 418 concurrent externalsgit-annexhttp://git-annex.branchable.com/devblog/day_418__concurrent_externals/git-annexikiwiki2016-10-05T17:31:23ZQuestionhttp://git-annex.branchable.com/devblog/day_418__concurrent_externals/comment_1_75e6569b143cf3f595ba9b356415a747/Adam2016-10-04T05:27:31Z2016-10-04T05:27:31Z
<p>Hi Joey,</p>
<p>Does this mean that, for example, when uploading many small files to a special remote, that git-annex could upload more than one at a time? Sometimes it seems like the time spent waiting for a small file to upload is mostly spent waiting for the server to begin accepting the file. With lots of small files, this adds up (similar to how round-trip time slows down Obnam significantly when using a network repo).</p>
<p>Thanks.</p>
comment 2http://git-annex.branchable.com/devblog/day_418__concurrent_externals/comment_2_067909ed54f3f9295f90233f25ccfd70/joey2016-10-04T15:30:46Z2016-10-04T15:16:01Z
<p>Well yes, git-annex -J4 or something can parallelize any operation and
avoid such latencies.</p>
comment 3http://git-annex.branchable.com/devblog/day_418__concurrent_externals/comment_3_39c729fc2ea4b597d13e54d60bb36d0d/Adam2016-10-04T21:24:50Z2016-10-04T21:24:50Z
Oh, I see, thanks. Have you thought about adding an option for that to the assistant? For example, I tested syncing a bunch of text files through a special remote, and even though the files were small, the per-command latency added up, so the effective transfer rate wasn't very high. If it were easy to set the concurrency option in the assistant, I guess it would help with that.
bittorrenthttp://git-annex.branchable.com/devblog/day_418__concurrent_externals/comment_4_6b254fbe92d40abc30d01afe5d039afc/evilmoo2016-10-05T03:53:05Z2016-10-05T03:53:05Z
Will this speed up the problems with added Bittorrent URLs only downloading one file at a time?
comment 5http://git-annex.branchable.com/devblog/day_418__concurrent_externals/comment_5_5e851b62bdca1cb8d8062d4890c77956/joey2016-10-05T17:31:23Z2016-10-05T15:31:37Z
@evilmoo is that problem detailed somewhere?
comment 6http://git-annex.branchable.com/devblog/day_418__concurrent_externals/comment_6_ea5d42bcb03af937a604119039f9073a/joey2016-10-05T17:31:23Z2016-10-05T15:32:15Z
@Adam adding a concurrency setting to the assistant would indeed be nice.