yt-dlp: parse/handle (error) "Video unavailable"

ATM git-annex addurl would happily download HTML from youtube instead of errorring, which would probably a better behavior, if video was removed from YouTube or made private. E.g.

here is a detailed git annex --debug addurl output

$> git annex --debug addurl --file Комендантські_балачки/2024-01-28-___Стрім_20_30_28.01.2024_чат_рулетка_LIVE_стрим._Андрій_Попик.mkv https://www.youtube.com/watch?v=jy01CnsQ9ec
[2024-01-29 16:55:36.511153059] (Utility.Process) process [2205617] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","show-ref","git-annex"]
[2024-01-29 16:55:36.515415282] (Utility.Process) process [2205617] done ExitSuccess
[2024-01-29 16:55:36.515888181] (Utility.Process) process [2205618] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","show-ref","--hash","refs/heads/git-annex"]
[2024-01-29 16:55:36.519652103] (Utility.Process) process [2205618] done ExitSuccess
[2024-01-29 16:55:36.521989271] (Utility.Process) process [2205619] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","cat-file","--batch"]
addurl https://www.youtube.com/watch?v=jy01CnsQ9ec
[2024-01-29 16:55:36.546921281] (Utility.Url) Request {
  host                 = "www.youtube.com"
  port                 = 443
  secure               = True
  requestHeaders       = [("Accept-Encoding",""),("User-Agent","git-annex/10.20231227-1~ndall+1")]
  path                 = "/watch"
  queryString          = "?v=jy01CnsQ9ec"
  method               = "HEAD"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = ResponseTimeoutDefault
  requestVersion       = HTTP/1.1
  proxySecureMode      = ProxySecureWithConnect
}

[2024-01-29 16:55:36.767345074] (Utility.Process) process [2205641] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","symbolic-ref","-q","HEAD"]
[2024-01-29 16:55:36.771989935] (Utility.Process) process [2205641] done ExitSuccess
[2024-01-29 16:55:36.772619847] (Utility.Process) process [2205642] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","show-ref","refs/heads/master"]
[2024-01-29 16:55:36.777260151] (Utility.Process) process [2205642] done ExitSuccess
[2024-01-29 16:55:36.778775886] (Utility.Process) process [2205643] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","check-attr","-z","--stdin","annex.backend","annex.largefiles","annex.numcopies","annex.mincopies","--"]
[2024-01-29 16:55:36.78290662] (Utility.Url) Request {
  host                 = "www.youtube.com"
  port                 = 443
  secure               = True
  requestHeaders       = [("Accept-Encoding","identity"),("User-Agent","git-annex/10.20231227-1~ndall+1")]
  path                 = "/watch"
  queryString          = "?v=jy01CnsQ9ec"
  method               = "GET"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = ResponseTimeoutDefault
  requestVersion       = HTTP/1.1
  proxySecureMode      = ProxySecureWithConnect
}

[2024-01-29 16:55:36.982919787] (Utility.Process) process [2205644] read: yt-dlp ["https://www.youtube.com/watch?v=jy01CnsQ9ec","--get-filename","--no-warnings","--no-playlist"]
[2024-01-29 16:55:38.035161914] (Utility.Process) process [2205644] done ExitFailure 1
[2024-01-29 16:55:38.036097212] (Utility.Process) p(to Комендантські_балачки/2024-01-28-___Стрім_20_30_28.01.2024_чат_рулетка_LIVE_стрим._Андрій_Попик.mkv) "--stdin","--verbose","--non-matching"]
[2024-01-29 16:55:38.054030685] (Messages.explain) [ Комендантські_балачки/2024-01-28-___Стрім_20_30_28.01.2024_чат_рулетка_LIVE_стрим._Андрій_Попик.mkv does not match annex.largefiles: mimeencoding=binary[FALSE] ]
                                                                                                                                                            (non-large file; adding content to git repository)
[2024-01-29 16:55:38.054895156] (Utility.Process) process [2205761] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","hash-object","-w","--no-filters","--stdin-paths"]
[2024-01-29 16:55:38.072695867] (Utility.Process) process [2205762] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","-c","filter.annex.smudge=","-c","filter.annex.clean=","-c","filter.annex.process=","write-tree"]
[2024-01-29 16:55:38.078372317] (Utility.Process) process [2205762] done ExitSuccess
[2024-01-29 16:55:38.07887887] (Utility.Process) process [2205763] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","show-ref","--hash","refs/annex/last-index"]
addurl https://www.youtube.com/watch?v=jy01CnsQ9ec (to Комендантські_балачки/2024-01-28-___Стрім_20_30_28.01.2024_чат_рулетка_LIVE_стрим._Андрій_Попик.mkv) (non-large file; adding content to git repository) ok
(recording state in git...)
[2024-01-29 16:55:38.083368582] (Utility.Process) process [2205764] feed: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","update-index","-z","--index-info"]
[2024-01-29 16:55:38.129969795] (Utility.Process) process [2205764] done ExitSuccess
[2024-01-29 16:55:38.130183526] (Database.Handle) commitDb start
[2024-01-29 16:55:38.130668768] (Database.Handle) commitDb done
[2024-01-29 16:55:38.131339954] (Utility.Process) process [2205619] done ExitSuccess
[2024-01-29 16:55:38.131872474] (Utility.Process) process [2205643] done ExitSuccess
[2024-01-29 16:55:38.132363417] (Utility.Process) process [2205761] done ExitSuccess
[2024-01-29 16:55:38.132786755] (Utility.Process) process [2205760] done ExitFailure 1

$> file -L Комендантські_балачки/2024-01-28-___Стрім_20_30_28.01.2024_чат_рулетка_LIVE_стрим._Андрій_Попик.mkv
Комендантські_балачки/2024-01-28-___Стрім_20_30_28.01.2024_чат_рулетка_LIVE_стрим._Андрій_Попик.mkv: HTML document, ASCII text, with very long lines (56909)

whenever that video is now announced to be Private if you go to that youtube url or yt-dlp saying that

❯ yt-dlp https://www.youtube.com/watch?v=jy01CnsQ9ec --get-filename --no-warnings --no-playlist
ERROR: [youtube] jy01CnsQ9ec: Video unavailable. This video has been removed by the uploader
❯ yt-dlp --version
2023.11.16

note that if I fake up that youtube url by changing letter to the next d - response does not include any reason

❯ yt-dlp https://www.youtube.com/watch?v=jy01CnsQ9ed --get-filename --no-warnings --no-playlist
ERROR: [youtube] jy01CnsQ9ed: Video unavailable

I don't consider this a bug because with --no-raw it will do what you want. done --Joey

RSS Atom

comment 1

To get the behavior you want, use --no-raw:

git-annex addurl --no-raw 'https://www.youtube.com/watch?v=jy01CnsQ9ec'
addurl https://www.youtube.com/watch?v=jy01CnsQ9ec

  Unable to use youtube-dl or a special remote and --no-raw was specified: no media in url
failed
addurl: 1 failed

Bear in mind that yt-dlp supports many websites, not just youtube, and youtube probably has many domains (eg youtu.be). And addurl can legitimately be used when you want to archive the current content of an url. So in this case git-annex can't distinguish between a webpage with a withdrawn or private video and any other url that a user would expect it to download.

Comment by joey — Tue Jan 30 19:43:50 2024

Remove comment