get from the web remote fails (redirect?) - lack of further debug info

Please describe the problem.

Originally reported on https://github.com/datalad/datalad/issues/2472

First I present full initial dump, then minimal set of commands and a few observations (e.g. seems to be related to running it within that docker environment, but cannot figure out why)

$> docker run -it --rm miykael/datalad_test
neuro@d0008523faf7:~$ cd ds000114/derivatives/fmriprep/
neuro@d0008523faf7:~/ds000114/derivatives/fmriprep$ git annex get --debug sub-01.html
[2018-05-06 15:30:05.79497316] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","ls-files","--cached","-z","--","sub-01.html"]
get sub-01.html [2018-05-06 15:30:05.801191505] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","git-annex"]
[2018-05-06 15:30:05.828819907] process done ExitSuccess
[2018-05-06 15:30:05.82896273] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","--hash","refs/heads/git-annex"]
[2018-05-06 15:30:05.832716117] process done ExitSuccess
[2018-05-06 15:30:05.833138385] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","log","refs/heads/git-annex..7d2ed09660cc91efbeb52b5792fb372622463813","--pretty=%H","-n1"]
[2018-05-06 15:30:05.837375012] process done ExitSuccess
[2018-05-06 15:30:05.837896891] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch"]
[2018-05-06 15:30:05.838257537] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch-check=%(objectname) %(objecttype) %(objectsize)"]
(from web...) 

(from web...) 


  Unable to access these remotes: web

  Try making some of these repositories available:
    00000000-0000-0000-0000-000000000001 -- web
    b8b7bab3-c978-4f50-8ed4-e77900198eb9 -- yoh@smaug:/mnt/btrfs/datasets/datalad/crawl/workshops/nih-workshop-2017/ds000114/derivatives/fmriprep
failed
[2018-05-06 15:30:05.877509068] process done ExitSuccess
[2018-05-06 15:30:05.877986803] process done ExitSuccess
git-annex: get: 1 failed

neuro@d0008523faf7:~/ds000114/derivatives/fmriprep$ git annex whereis sub-01.html
whereis sub-01.html (2 copies) 
    00000000-0000-0000-0000-000000000001 -- web
    b8b7bab3-c978-4f50-8ed4-e77900198eb9 -- yoh@smaug:/mnt/btrfs/datasets/datalad/crawl/workshops/nih-workshop-2017/ds000114/derivatives/fmriprep

  web: https://dl.dropbox.com/s/f03sxu7omhsibyc/sub-01.html?dl=0
ok

neuro@d0008523faf7:~/ds000114/derivatives/fmriprep$ curl https://dl.dropbox.com/s/f03sxu7omhsibyc/sub-01.html?dl=0
<html>
  <head><title>Found</title></head>
  <body>
    <h1>Found</h1>
    <p>The resource was found at <a href="https://dl.dropboxusercontent.com/s/f03sxu7omhsibyc/sub-01.html?dl=0">https://dl.dropboxusercontent.com/s/f03sxu7omhsibyc/sub-01.html?dl=0</a>;
you should be redirected automatically.

<!--  --></p>
    <hr noshade>
    <div align="right">WSGI Server</div>
  </body>
</html>

neuro@d0008523faf7:~/ds000114/derivatives/fmriprep$ git annex version
git-annex version: 6.20180416+gitg86b18966f-1~ndall+1
build flags: Assistant Webapp Pairing S3(multipartupload)(storageclasses) WebDAV Inotify DBus DesktopNotify ConcurrentOutput TorrentParser MagicMime Feeds Testsuite
dependency versions: aws-0.14.1 bloomfilter-2.0.1.0 cryptonite-0.20 DAV-1.3.1 feed-0.3.11.1 ghc-8.0.1 http-client-0.4.31.1 persistent-sqlite-2.6 torrent-10000.0.0 uuid-1.3.12 yesod-1.4.3
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 SHA1E SHA1 MD5E MD5 WORM URL
remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar hook external
local repository version: 5
supported repository versions: 3 5 6
upgrade supported from repository versions: 0 1 2 3 4 5
operating system: linux x86_64

Minimal set of commands to replicate

git clone http://datasets.datalad.org/workshops/nih-2017/ds000114/derivatives/fmriprep/.git
cd fmriprep
git annex get --debug --from=web sub-01.html

I think the issue is somehow specific to running it within that docker -- if those commands ran on laptop, it works just fine!
Issue persists with the 6.20180501-g0d2527e67 manually installed in the container
Since then I also provided that file from the webserver, so if you do git fetch, it would start fetching that file from the web server (thus obfuscating the issue), so you would need ' --from=web ' to force an attempt to download from the web
if before, with invoking wget/curl commands it was possible to see what is going on, ATM --debug of git-annex does not give any information about what is attempted to be fetched from web remote and what problem it is experiencing. I think it would be very valuable to get more troubleshooting information output in --debug

This does not involve redirects. It's hitting the url a couple of times on failure, thus the multiple "from web". That was due to a bug in the forward retry code; Just 0 > Nothing. Fixed.

I added a display of the error message from the web server.

fixed --Joey