Currently git-annex uses wget and curl for downloading urls. Which is used depends on the situation, since both have their limitations and quirks.

This often confuses users, who expect annex.web-options to only apply to whichever program git-annex was running, and put in an option that breaks the other program. Or, configure a netrc file, which wget uses by default, but curl does not.

Also, using these external programs avoids keeping a http connection open and pipelining requests, so it makes mass url downloads a lot slower than if git-annex used http-conduit to do url downloads itself. yoh has requested http pipelining.

(git-annex was creating a new http manager each time it hit an url, except for in the S3 remote which reused a single manager. That's now been improved, so all http-conduit use in git-annex reuses a http manager, and so will do http pipelining.)

For file: ftp: and more unusual urls, http-conduit can't support them. git-annex does support those urls, and people rely on that, so it would still need to use wget or curl for those.

wget is also not shipped with git-annex on Windows or OSX, only curl is, and it would be good to only use one of the programs, not both, when handing those unusual urls.

See also, support .netrc for fsck --from web. That some users rely on git-annex using wget and a netrc file is kind of problimatic if switching to http-conduit which does not support it. Maybe require users to set annex.web-download-command if they want to make it use something that supports netrc?


Implemented Utility.Url.downloadC that is the (nontrivial) download a file with resume support using http-conduit. It falls back to curl to handle urls that http-conduit does not support. Now we only have to decide what to do about the above edge cases..

Let's drop use of wget entirely, as it was only using it because I preferred wget's progress bar to curl's. The user can still force wget with annex.web-download-command.

That leaves users who have a .netrc file or want to use annex.web-options. Since curl requires --netrc in order to use the .netrc file, require users who want to use the .netrc to set "annex.web-options = --netrc". When "annex.web-options" is set, always use curl (unless overridden by annex.web-download-command). Otherwise, use conduit.

done --Joey