Realized recently that despite all the nice concurrency support in git-annex, external special remotes were limited to handling one request at a time.
While the external special remote prococol could almost support concurrent requests, that would complicate implementing them, and probably need a version flag to enable to avoid breaking existing ones.
Instead, made git-annex start up multiple external special remote processes as needed to handle concurrency.
Today's work was sponsored by Josh Taylor on Patreon.
Hi Joey,
Does this mean that, for example, when uploading many small files to a special remote, that git-annex could upload more than one at a time? Sometimes it seems like the time spent waiting for a small file to upload is mostly spent waiting for the server to begin accepting the file. With lots of small files, this adds up (similar to how round-trip time slows down Obnam significantly when using a network repo).
Thanks.
Well yes, git-annex -J4 or something can parallelize any operation and avoid such latencies.