fails to get from apache2 server URL backend file

Please describe the problem.

For a relaxed url youtube video, git-annex seems just completely skip even trying (I see no apache2 log hits) to download from http git remote where it even points to correct HTTP address, and then just proceeds to yt-dlp to just fail there:

❯ git clone https://datasets.datalad.org/repronim/ReproTube/AFNIBootcamp/.git/
Cloning into 'AFNIBootcamp'...
remote: Enumerating objects: 5904, done.
remote: Counting objects: 100% (5904/5904), done.
remote: Compressing objects: 100% (1793/1793), done.
remote: Total 5904 (delta 2659), reused 5554 (delta 2644), pack-reused 0 (from 0)
Receiving objects: 100% (5904/5904), 743.23 KiB | 2.23 MiB/s, done.
Resolving deltas: 100% (2659/2659), done.
❯ cd AFNIBootcamp
authors.tsv@  channel.json  channel_avatar.jpg@  playlists/  videos/
❯ git annex whereis videos/2020/04/2020-04-17_AFNI-Academy-AFNI-GUI-Clusterizing/video.mkv
whereis videos/2020/04/2020-04-17_AFNI-Academy-AFNI-GUI-Clusterizing/video.mkv (3 copies) 
    00000000-0000-0000-0000-000000000001 -- web
    cc815e85-73bc-4a5c-81c3-81a39b0c677b -- yoh@falkor:/srv/datasets.datalad.org/www/repronim/ReproTube/AFNIBootcamp [origin]
    f574aace-b921-4987-b376-f43cfcc0e925 -- annextube YouTube archive

  web: https://www.youtube.com/watch?v=3ZXfZfnRfyM
ok
❯ git annex --debug get --from origin videos/2020/04/2020-04-17_AFNI-Academy-AFNI-GUI-Clusterizing/video.mkv
[2026-02-12 17:58:20.476127402] (Utility.Process) process [1348659] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","show-ref","git-annex"]
[2026-02-12 17:58:20.477586712] (Utility.Process) process [1348659] done ExitSuccess
[2026-02-12 17:58:20.477947473] (Utility.Process) process [1348660] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","show-ref","--hash","refs/heads/git-annex"]
[2026-02-12 17:58:20.479504195] (Utility.Process) process [1348660] done ExitSuccess
[2026-02-12 17:58:20.480128621] (Utility.Process) process [1348661] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","log","refs/heads/git-annex..164c7074ef367be9c939366c3febb2322f70c103","--pretty=%H","-n1"]
[2026-02-12 17:58:20.482481122] (Utility.Process) process [1348661] done ExitSuccess
[2026-02-12 17:58:20.484072231] (Utility.Process) process [1348662] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","cat-file","--batch"]
[2026-02-12 17:58:20.488013705] (Utility.Process) process [1348663] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","ls-files","--stage","-z","--error-unmatch","--","videos/2020/04/2020-04-17_AFNI-Academy-AFNI-GUI-Clusterizing/video.mkv"]
[2026-02-12 17:58:20.488431021] (Utility.Process) process [1348664] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","cat-file","--batch-check=%(objectname) %(objecttype) %(objectsize)","--buffer"]
[2026-02-12 17:58:20.488864415] (Utility.Process) process [1348665] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","cat-file","--batch=%(objectname) %(objecttype) %(objectsize)","--buffer"]
[2026-02-12 17:58:20.489285814] (Utility.Process) process [1348666] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","cat-file","--batch=%(objectname) %(objecttype) %(objectsize)","--buffer"]
[2026-02-12 17:58:20.491835062] (Utility.Process) process [1348666] done ExitSuccess
[2026-02-12 17:58:20.491913957] (Utility.Process) process [1348665] done ExitSuccess
[2026-02-12 17:58:20.491944604] (Utility.Process) process [1348664] done ExitSuccess
[2026-02-12 17:58:20.491970167] (Utility.Process) process [1348663] done ExitSuccess
get videos/2020/04/2020-04-17_AFNI-Academy-AFNI-GUI-Clusterizing/video.mkv (from origin...) 
[2026-02-12 17:58:20.516237522] (Utility.Url) https://datasets.datalad.org/repronim/ReproTube/AFNIBootcamp/.git//annex/objects/zZ/3v/URL--yt&chttps&c%%www.youtube.com%watch,63v,613ZXfZfnRfyM/URL--yt&chttps&c%%www.youtube.com%watch,63v,613ZXfZfnRfyM
[2026-02-12 17:58:20.519744566] (Utility.Url) https://datasets.datalad.org/repronim/ReproTube/AFNIBootcamp/.git//annex/objects/950/20d/URL--yt&chttps&c%%www.youtube.com%watch,63v,613ZXfZfnRfyM/URL--yt&chttps&c%%www.youtube.com%watch,63v,613ZXfZfnRfyM
  download failed: invalid url

  failed to download content
(Delaying 1s before retrying....)
[2026-02-12 17:58:21.524457718] (Utility.Url) https://datasets.datalad.org/repronim/ReproTube/AFNIBootcamp/.git//annex/objects/zZ/3v/URL--yt&chttps&c%%www.youtube.com%watch,63v,613ZXfZfnRfyM/URL--yt&chttps&c%%www.youtube.com%watch,63v,613ZXfZfnRfyM
[2026-02-12 17:58:21.527050375] (Utility.Url) https://datasets.datalad.org/repronim/ReproTube/AFNIBootcamp/.git//annex/objects/950/20d/URL--yt&chttps&c%%www.youtube.com%watch,63v,613ZXfZfnRfyM/URL--yt&chttps&c%%www.youtube.com%watch,63v,6get videos/2020/04/2020-04-17_AFNI-Academy-AFNI-GUI-Clusterizing/video.mkv (from origin...) 
  download failed: invalid url

  failed to download content
(Delaying 1s before retrying....)

  download failed: invalid url

  failed to download content
(Delaying 2s before retrying....)

  download failed: invalid url

  failed to download content
failed
[2026-02-12 17:58:23.537290686] (Utility.Process) process [1348662] done ExitSuccess
get: 1 failed

for a simpler file -- works fine

❯ git annex whereis channel_avatar.jpg
whereis channel_avatar.jpg (2 copies) 
    cc815e85-73bc-4a5c-81c3-81a39b0c677b -- yoh@falkor:/srv/datasets.datalad.org/www/repronim/ReproTube/AFNIBootcamp [origin]
    f574aace-b921-4987-b376-f43cfcc0e925 -- annextube YouTube archive
ok
❯ git annex get --from origin channel_avatar.jpg
get channel_avatar.jpg (from origin...) ok
(recording state in git...)
❯ ls -l channel_avatar.jpg
lrwxrwxrwx 1 yoh yoh 196 Feb 12 17:57 channel_avatar.jpg -> .git/annex/objects/54/77/SHA256E-s107998--454529608f75da5804000d74018ff790ec24a03eef3544fc44c28071e31acd15.jpg/SHA256E-s107998--454529608f75da5804000d74018ff790ec24a03eef3544fc44c28071e31acd15.jpg

What steps will reproduce the problem?

  git clone https://datasets.datalad.org/repronim/ReproTube/AFNIBootcamp/.git/
  cd AFNIBootcamp
  git annex whereis videos/2020/04/2020-04-17_AFNI-Academy-AFNI-GUI-Clusterizing/video.mkv
  git annex --debug get --from origin videos/2020/04/2020-04-17_AFNI-Academy-AFNI-GUI-Clusterizing/video.mkv

What version of git-annex are you using? On what operating system?

❯ git annex version
git-annex version: 10.20250929-gf014fd60d05a3407e2f747e0394997d3780eeafc

but did try even most recent

Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)

when I was lucky - yes.

RSS Atom

the fix

TL;DR, the patch

diff --git a/Remote/Git.hs b/Remote/Git.hs
index 6b7dc77d98..4faaea082d 100644
--- a/Remote/Git.hs
+++ b/Remote/Git.hs
@@ -482,7 +482,12 @@ inAnnex' repo rmt st@(State connpool duc _ _ _ _) key
 keyUrls :: GitConfig -> Git.Repo -> Remote -> Key -> [String]
 keyUrls gc repo r key = map tourl locs'
   where
-   tourl l = Git.repoLocation repo ++ "/" ++ l
+   tourl l = Git.repoLocation repo ++ "/" ++ escapeURIString escchar l
+   -- Escape characters that are not allowed unescaped in a URI
+   -- path component, but don't escape '/' since the location
+   -- is a path with multiple components.
+   escchar '/' = True
+   escchar c = isUnescapedInURIComponent c
    -- If the remote is known to not be bare, try the hash locations
    -- used for non-bare repos first, as an optimisation.
    locs

seems to work well. Built in https://github.com/datalad/git-annex/pull/251 (CI tests still run), tested locally:

❯ /usr/bin/git-annex version
git-annex version: 10.20260115+git119-g43a3f3aaf2-1~ndall+1
build flags: Assistant Webapp Inotify DBus DesktopNotify TorrentParser MagicMime Benchmark Feeds Testsuite S3 WebDAV Servant
dependency versions: aws-0.24.1 bloomfilter-2.0.1.2 crypton-0.34 DAV-1.3.4 feed-1.3.2.1 ghc-9.6.6 http-client-0.7.17 torrent-10000.1.3 uuid-1.3.15 yesod-1.6.2.1
...
❯ /usr/bin/git-annex get --from origin video.mkv
get video.mkv (from origin...) ok
(recording state in git...)

to work. Here is claude's analysis which lead it to the fix:

  Bug Analysis: fails_to_get_from_apache2_server_URL_backend_file

  Root Cause

  The bug is in Remote/Git.hs:485 — the keyUrls function constructs URLs by simple string concatenation without URL-encoding the path components:

  tourl l = Git.repoLocation repo ++ "/" ++ l

  How the failure occurs

  1. Key: URL--yt:https://www.youtube.com/watch,63v,613ZXfZfnRfyM
  2. keyFile encoding (Annex/Locations.hs:783-795) converts : → &c and / → %:
  URL--yt&chttps&c%%www.youtube.com%watch,63v,613ZXfZfnRfyM
  3. keyUrls concatenates this directly into the URL path:
  https://datasets.datalad.org/.../.git//annex/objects/zZ/3v/URL--yt&chttps&c%%www.youtube.com%watch,63v,613ZXfZfnRfyM/...
  4. parseURIRelaxed (Utility/Url/Parse.hs:45-47) tries to parse this URL. It calls escapeURIString isAllowedInURI first, but % is allowed in URIs (it's the percent-encoding introducer), so it passes through unescaped.
  5. parseURI then sees %%w and %wa which are invalid percent-encoding sequences (% must be followed by two hex digits). The parse fails, returning Nothing.
  6. download' (Utility/Url.hs:389-391) hits the Nothing branch and returns "invalid url".

  Why SHA256E keys work

  SHA256E keys (like SHA256E-s107998--4545...) contain only alphanumeric characters, -, and .. The keyFile encoding produces no % or & characters, so the concatenated URL is always valid.

  The fix

  keyUrls in Remote/Git.hs:485 needs to URL-encode the path components. Other remotes already do this:

  - S3 (Remote/S3.hs:1221-1229): uses escapeURIString with a custom predicate keeping / but encoding everything else
  - WebDAV (Remote/WebDAV/DavLocation.hs:35): uses escapeURIString isUnescapedInURI

Comment by yarikoptic — Fri Feb 13 01:43:15 2026

Add a comment