tries to download a .mkv video without yt-dlp

Please describe the problem.

For some not obvious to me reason, git-annex used yt-dlp fine for many files but then does not use for another, very odd:

here is two files from (pushed all to http://github.com/TrueTube/Andriy_Popik)

(git)smaug:/tmp/Andriy_Popyk/Andriy_Popyk[master]git
$> git annex whereis Чат_рулетка/2024-09-30-Зеки_-_основа_россии._Андрій_Попик.mkv Чат_рулетка/2024-10-01-Рассеянка__Андрій_Попик.mkv
whereis "\320\247\320\260\321\202_\321\200\321\203\320\273\320\265\321\202\320\272\320\260/2024-09-30-\320\227\320\265\320\272\320\270_-_\320\276\321\201\320\275\320\276\320\262\320\260_\321\200\320\276\321\201\321\201\320\270\320\270._\320\220\320\275\320\264\321\200\321\226\320\271_\320\237\320\276\320\277\320\270\320\272.mkv" (2 copies) 
    00000000-0000-0000-0000-000000000001 -- web
    05689e10-d659-4239-a614-a6de95b11208 -- yoh@smaug:/mnt/btrfs/datasets/datalad/crawl-misc/TrueTube/popik

  web: https://www.youtube.com/watch?v=Ikn4LnYDcKU
ok
whereis "\320\247\320\260\321\202_\321\200\321\203\320\273\320\265\321\202\320\272\320\260/2024-10-01-\320\240\320\260\321\201\321\201\320\265\321\217\320\275\320\272\320\260__\320\220\320\275\320\264\321\200\321\226\320\271_\320\237\320\276\320\277\320\270\320\272.mkv" (1 copy) 
    00000000-0000-0000-0000-000000000001 -- web

  web: https://www.youtube.com/watch?v=5rFP3DlC2MU
ok

for one ($> git annex get --debug Чат_рулетка/2024-09-30-Зеки_-_основа_россии._Андрій_Попик.mkv) works fine -- starts downloading using yt-dlp. But for the other:

$> git annex get --debug Чат_рулетка/2024-10-01-Рассеянка__Андрій_Попик.mkv           
[2024-10-01 17:00:12.822269342] (Utility.Process) process [2784960] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","show-ref","git-annex"]
[2024-10-01 17:00:12.827363166] (Utility.Process) process [2784960] done ExitSuccess
[2024-10-01 17:00:12.828029045] (Utility.Process) process [2784961] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","show-ref","--hash","refs/heads/git-annex"]
[2024-10-01 17:00:12.832174221] (Utility.Process) process [2784961] done ExitSuccess
[2024-10-01 17:00:12.832951895] (Utility.Process) process [2784962] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","log","refs/heads/git-annex..1c3a3eacf78934c71cdf560061207cdb13422fd0","--pretty=%H","-n1"]
[2024-10-01 17:00:12.835645728] (Utility.Process) process [2784962] done ExitSuccess
[2024-10-01 17:00:12.840540312] (Utility.Process) process [2784963] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","cat-file","--batch"]
[2024-10-01 17:00:12.855549135] (Utility.Process) process [2784964] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","ls-files","--stage","-z","--error-unmatch","--","\1063\1072\1090_\1088\1091\1083\1077\1090\1082\1072/2024-10-01-\1056\1072\1089\1089\1077\1103\1085\1082\1072__\1040\1085\1076\1088\1110\1081_\1055\1086\1087\1080\1082.mkv"]
[2024-10-01 17:00:12.856270222] (Utility.Process) process [2784965] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","cat-file","--batch-check=%(objectname) %(objecttype) %(objectsize)","--buffer"]
[2024-10-01 17:00:12.85694905] (Utility.Process) process [2784966] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","cat-file","--batch=%(objectname) %(objecttype) %(objectsize)","--buffer"]
[2024-10-01 17:00:12.857415068] (Utility.Process) process [2784967] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","cat-file","--batch=%(objectname) %(objecttype) %(objectsize)","--buffer"]
[2024-10-01 17:00:12.864931666] (Utility.Process) process [2784967] done ExitSuccess
[2024-10-01 17:00:12.865042169] (Utility.Process) process [2784966] done ExitSuccess
[2024-10-01 17:00:12.865124487] (Utility.Process) process [2784965] done ExitSuccess
[2024-10-01 17:00:12.865195432] (Utility.Process) process [2784964] done ExitSuccess
get "\320\247\320\260\321\202_\321\200\321\203\320\273\320\265\321\202\320\272\320\260/2024-10-01-\320\240\320\260\321\201\321\201\320\265\321\217\320\275\320\272\320\260__\320\220
\320\275\320\264\321\200\321\226\320\271_\320\237\320\276\320\277\320\270\320\272.mkv" (from web...) 
[2024-10-01 17:00:12.898739526] (Utility.Url) Request {
  host                 = "www.youtube.com"
  port                 = 443
  secure               = True
  requestHeaders       = [("Accept-Encoding","identity"),("User-Agent","git-annex/10.20240927+git1-gf3403e9691-1~ndall+1")]
  path                 = "/watch"
  queryString          = "?v=5rFP3DlC2MU"
  method               = "GET"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = ResponseTimeoutDefault
  requestVersion       = HTTP/1.1
  proxySecureMode      = ProxySecureWithConnect
get "\320\247\320\260\321\202_\321\200\321\203\320\273\320\265\321\202\320\272\320\260/2024-10-01-\320\240\320\260\321\201\321\201\320\265\321\217\320\275\320\272\320\260__\320\220\320\275\320\264\321\200\321\226\320\271_\320\237\320\276\320\277\320\270\320\272.mkv" (from web...) 
  Verification of content failed

  Unable to access these remotes: web

  No other repository is known to contain the file.

  (Note that these git remotes have annex-ignore set: origin)
failed
[2024-10-01 17:00:13.516843908] (Utility.Process) process [2784963] done ExitSuccess
get: 1 failed

What steps will reproduce the problem?

git clone http://github.com/TrueTube/Andriy_Popik
cd Andriy*
git config annex.security.allowed-ip-addresses all
git annex get --debug Чат_рулетка/2024-10-01-Рассеянка__Андрій_Попик.mkv

What version of git-annex are you using? On what operating system?

$> git annex version                                                       
git-annex version: 10.20240927+git1-gf3403e9691-1~ndall+1

RSS Atom

comment 1

I suspect that one file has a "yt:" url recorded and the other one does not. That's not displayed by whereis but you can see it if you look at the $key.log.web file in the git-annex branch.

That typically would happen if you used git-annex addurl --raw, or added the url when yt-dlp was not working, although that doesn't explain how the youtube title got into the filename. (Maybe importfeed?)

(I seemed to remember you having a similar problem before, and found https://git-annex.branchable.com/todo/yt-dlp__58___parse__47__handle___40__error__41_____34__Video_unavailable__34__/ which seems possibly relevant to how this could have happened.)

Comment by joey — Mon Oct 21 15:50:26 2024

FTR commands to check and "fix up"

in fears against modification of files in git-annex branch directly, here is the commands to 'check'

$> f=Чат_рулетка/2026-03-05-_армянин_за_путина._Армянин_из_россии_Воевал_против_Украины.mkv
$> key=$(readlink "$f" | xargs basename); alog=$(git ls-tree -r git-annex | grep "$key" | awk '/.web$/{print $4;}'); git show "git-annex:$alog"
1772708470s 1 https://www.youtube.com/watch?v=0fcKYGsBZxU

First I tried to fix via re-addurl, and we do get some difference:

$> git rm "$f"; git annex addurl --no-raw --file "$f" "$url" rm 'Чатрулетка/2026-03-05-армянинзапутина.АрмянинизроссииВоевалпротивУкраины.mkv' addurl https://www.youtube.com/watch?v=0fcKYGsBZxU (using yt-dlp) (to Чатрулетка/2026-03-05-армянинзапутина.АрмянинизроссииВоевалпротивУкраины.mkv) ok (recording state in git...) $> git status On branch master Your branch is up to date with 'origin/master'.

Changes to be committed: (use "git restore --staged ..." to unstage) modified: Чатрулетка/2026-03-05-армянинзапутина.АрмянинизроссииВоевалпротивУкраины.mkv

$> git diff --cached diff --git a/Чатрулетка/2026-03-05-армянинзапутина.АрмянинизроссииВоевалпротивУкраины.mkv b/Чатрулетка/2026-03-05-армянинзапутина.АрмянинизроссииВоевалпротивУкраины.mkv index e59a58c35..e12bb1280 120000 --- a/Чатрулетка/2026-03-05-армянинзапутина.АрмянинизроссииВоевалпротивУкраины.mkv +++ b/Чатрулетка/2026-03-05-армянинзапутина.АрмянинизроссииВоевалпротивУкраины.mkv @@ -1 +1 @@ -../.git/annex/objects/KQ/x1/URL-s0--https&c%%www.youtube.com%watch,63v,610fcKYGsBZxU/URL-s0--https&c%%www.youtube.com%watch,63v,610fcKYGsBZxU \ No newline at end of file +../.git/annex/objects/wq/jM/URL--yt&chttps&c%%www.youtube.com%watch,63v,610fcKYGsBZxU/URL--yt&chttps&c%%www.youtube.com%watch,63v,610fcKYGsBZxU \ No newline at end of file


for which I did not really care as long as I got that file if metadata transferred, but it didn't:

$> git commit -m 'redownloaded "unlucky" video for which no yt: was added' $f [master 379d379ea] redownloaded "unlucky" video for which no yt: was added 1 file changed, 1 insertion(+), 1 deletion(-) $> git annex metadata "$f" metadata Чатрулетка/2026-03-05-армянинзапутина.АрмянинизроссииВоевалпротивУкраины.mkv

ok ```

Also when I used recent after 202601 version which would auto-upgrade to use VURL key difference was to switch from URL to VURL. Could you please point me on where to read up on VURLs and their benefit for relaxed URLs?

then I tried to do the dance with unregisterurl, rmurl, addurl, which ended up having

$> key=$(readlink "$f" | xargs basename); alog=$(git ls-tree -r git-annex | grep "$key" | awk '/.web$/{print $4;}'); git show "git-annex:$alog"
1772750261s 0 https://www.youtube.com/watch?v=0fcKYGsBZxU
1772750309s 1 yt:https://www.youtube.com/watch?v=0fcKYGsBZxU

and for which I still was not able to get it:

$> git annex get "$f"
get Чат_рулетка/2026-03-05-_армянин_за_путина._Армянин_из_россии_Воевал_против_Украины.mkv (from web...)
  Verification of content failed

  Unable to access these remotes: web

  No other repository is known to contain the file.

  (Note that these git remotes have annex-ignore set: origin)
failed
get: 1 failed
git annex get "$f"  8.22s user 3.63s system 112% cpu 10.505 total

although I think it did fetch it. But i guess it is because of the -s0 in the original key! So original way with git rm + addurl was kinda legit as it also fixed up the URL BUT it lost the metadata for the key.

Is there a quick way to copy metadata from another key? (like internally it does for the same path?)

Or is there a better way to 'fix up URL/key' which would you recommend Joey so I could retain metadata?

Comment by yarikoptic — Thu Mar 5 22:45:31 2026