regression - yt: prefix for "regular" urls

Please describe the problem.

some tests started to fail in datalad when tried current master version of annex

What version of git-annex are you using? On what operating system?

6.20171205+gitgcf05ebbeb-1~ndall+1

Please provide any additional information below.

(git-annex)hopa:/tmp/datalad_temp_test_AnnexRepo_web_remoteRusdWA[master]git
$> git annex whereis --json 127.0.0.1_about.txt
{"command":"whereis","note":"web: http://example.com/someurl\nweb: yt:http://127.0.0.1:33587/about.txt\n","success":true,"untrusted":[],"key":"SHA256E-s14--198775960d0d9827f134da4e219e5793949d24b62bfcb92aa9e2dc16d411b776.txt","whereis":[{"here":false,"uuid":"00000000-0000-0000-0000-000000000001","urls":["http://example.com/someurl","yt:http://127.0.0.1:33587/about.txt"],"description":"web"}],"file":"127.0.0.1_about.txt"}

$> git annex whereis 127.0.0.1_about.txt      
whereis 127.0.0.1_about.txt (1 copy) 
    00000000-0000-0000-0000-000000000001 -- web

  web: http://example.com/someurl
  web: yt:http://127.0.0.1:33587/about.txt

This is a new behavior, this test was passing before, and I do not think we suggested that those test files are anyhow related to youtube before.

Some tests also failed related to our datalad archives git annex special remote but I didn't look into them yet -- I guess though that it is related ;-)

fixed --Joey

RSS Atom

comment 1

Reporoduced by putting "Lots of abouts" in a http://localhost/about.txt file, and running git annex addurl --file with an existing file and that url.

Comment by joey — Wed Dec 6 15:51:03 2017

Remove comment

comment 2

Comment by joey — Wed Dec 6 16:02:55 2017

Remove comment

comment 2

Thanks for the fix... seems to be a bit incomplete though -- in the --fast mode, url still has yt: prefix, reflected in the key as well:

$> ls -l
total 4
lrwxrwxrwx 1 yoh yoh 110 Dec  7 21:27 1-copy.dat -> .git/annex/objects/gw/pw/URL--yt&chttp&c%%127.0.0.1&c34337%d1%1.dat/URL--yt&chttp&c%%127.0.0.1&c34337%d1%1.dat

$> git annex whereis 1-copy.dat
whereis 1-copy.dat (1 copy) 
    00000000-0000-0000-0000-000000000001 -- web

  web: yt:http://127.0.0.1:34337/d1/1.dat
ok

$> git annex version
git-annex version: 6.20171206+gitgc6e4bc0a2-1~ndall+1

Comment by yarikoptic — Fri Dec 8 02:32:15 2017

Remove comment

comment 4

That one happened with git annex addurl --fast $url so a different code path. Had to add a html page check to youtubeDlFileName to fix it.

Comment by joey — Fri Dec 8 18:48:46 2017

Remove comment

comment 5

ok, tested with 6.20171208+gitg01f78e877-1~ndall+1 -- --fast mode issue is indeed resolved. Thanks! Failing tests now only relate to our special remote (datalad-archives) which interfaces urls, so I thought it might relate... I will try to distill more info tomorrow unless you beat me to it figuring out where that regression could be (upon quick look didn't spot any yt: anywhere, so probably another issue)

Comment by yarikoptic — Mon Dec 11 05:27:15 2017

Remove comment

comment 6

ok, first initial bit of information: there is a difference in interactions with the special remote now. Here is the diff between old and new runs (sorry - content was also changing so there is difference in keys as well):

$> diff -Nar -u6 /tmp/datalad_temp_tree_check_basic_scenarioVEOIhb/.git/bin/git-annex-remote-datalad-archive /tmp/datalad_temp_tree_check_basic_scenarioIfsLo7/.git/bin/git-annex-remote-datalad-archive
--- /tmp/datalad_temp_tree_check_basic_scenarioVEOIhb/.git/bin/git-annex-remote-datalad-archive 2017-12-11 08:56:38.381298365 -0500
+++ /tmp/datalad_temp_tree_check_basic_scenarioIfsLo7/.git/bin/git-annex-remote-datalad-archive 2017-12-11 08:56:14.885677071 -0500
@@ -48,66 +48,72 @@
 ### ['git', '-c', 'receive.autogc=0', '-c', 'gc.auto=0', 'annex', 'add', '--debug', '--json', 'simple.txt']
 
 ### ['git', '-c', 'receive.autogc=0', '-c', 'gc.auto=0', 'commit', '-m', 'Added the load file']
 
 ### ['git', '-c', 'receive.autogc=0', '-c', 'gc.auto=0', 'annex', 'lookupkey', '--debug', 'a.tar.gz']
 
-### ['git', '-c', 'receive.autogc=0', '-c', 'gc.auto=0', 'annex', 'addurl', '--debug', '--relaxed', '--file=simple.txt', 'dl+archive:SHA256E-s172--70cf6dd95738e5d3672a7139a2785b0a979f0f7955d0f6da0d94cc03c84a63b7.tar.gz#path=a/d/+%22%27%3Ba%26b%26cd+%60%7C+']
+### ['git', '-c', 'receive.autogc=0', '-c', 'gc.auto=0', 'annex', 'addurl', '--debug', '--relaxed', '--file=simple.txt', 'dl+archive:SHA256E-s173--db0a9680f8d15578de8e4a5b5c1e87f36d9372d6118fb24c1c60f390e71ad3c1.tar.gz#path=a/d/+%22%27%3Ba%26b%26cd+%60%7C+']
 #send VERSION 1
 #recv PREPARE
 #send PREPARE-SUCCESS
 #recv GETCOST
 #send COST 500
 #recv GETAVAILABILITY
 #send AVAILABILITY LOCAL
-#recv CLAIMURL dl+archive:SHA256E-s172--70cf6dd95738e5d3672a7139a2785b0a979f0f7955d0f6da0d94cc03c84a63b7.tar.gz#path=a/d/+%22%27%3Ba%26b%26cd+%60%7C+
-#send DEBUG Claiming url 'dl+archive:SHA256E-s172--70cf6dd95738e5d3672a7139a2785b0a979f0f7955d0f6da0d94cc03c84a63b7.tar.gz#path=a/d/+%22%27%3Ba%26b%26cd+%60%7C+'
+#recv CLAIMURL dl+archive:SHA256E-s173--db0a9680f8d15578de8e4a5b5c1e87f36d9372d6118fb24c1c60f390e71ad3c1.tar.gz#path=a/d/+%22%27%3Ba%26b%26cd+%60%7C+
+#send DEBUG Claiming url 'dl+archive:SHA256E-s173--db0a9680f8d15578de8e4a5b5c1e87f36d9372d6118fb24c1c60f390e71ad3c1.tar.gz#path=a/d/+%22%27%3Ba%26b%26cd+%60%7C+'
 #send CLAIMURL-SUCCESS
-#recv CHECKURL dl+archive:SHA256E-s172--70cf6dd95738e5d3672a7139a2785b0a979f0f7955d0f6da0d94cc03c84a63b7.tar.gz#path=a/d/+%22%27%3Ba%26b%26cd+%60%7C+
+#recv CHECKURL dl+archive:SHA256E-s173--db0a9680f8d15578de8e4a5b5c1e87f36d9372d6118fb24c1c60f390e71ad3c1.tar.gz#path=a/d/+%22%27%3Ba%26b%26cd+%60%7C+
 #send CHECKURL-CONTENTS UNKNOWN
 #recv 
 
 ### ['git', '-c', 'receive.autogc=0', '-c', 'gc.auto=0', 'annex', 'drop', '--debug', '--json', 'simple.txt']
 #send VERSION 1
 #recv PREPARE
 #send PREPARE-SUCCESS
 #recv CHECKPRESENT SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.txt
 #send GETURLS SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.txt dl+archive:
-#recv VALUE dl+archive:SHA256E-s172--70cf6dd95738e5d3672a7139a2785b0a979f0f7955d0f6da0d94cc03c84a63b7.tar.gz#path=a/d/+%22%27%3Ba%26b%26cd+%60%7C+
+#recv VALUE dl+archive:SHA256E-s173--db0a9680f8d15578de8e4a5b5c1e87f36d9372d6118fb24c1c60f390e71ad3c1.tar.gz#path=a/d/+%22%27%3Ba%26b%26cd+%60%7C+
 #recv VALUE 
 #send CHECKPRESENT-SUCCESS SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.txt
 #recv 
 
 ### ['git', '-c', 'receive.autogc=0', '-c', 'gc.auto=0', 'annex', 'whereis', '--debug', '--json', 'simple.txt']
 #send VERSION 1
 #recv PREPARE
 #send PREPARE-SUCCESS
 #recv WHEREIS SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.txt
 #send WHEREIS-FAILURE
-#recv CLAIMURL dl+archive:SHA256E-s172--70cf6dd95738e5d3672a7139a2785b0a979f0f7955d0f6da0d94cc03c84a63b7.tar.gz#path=a/d/+%22%27%3Ba%26b%26cd+%60%7C+
-#send DEBUG Claiming url 'dl+archive:SHA256E-s172--70cf6dd95738e5d3672a7139a2785b0a979f0f7955d0f6da0d94cc03c84a63b7.tar.gz#path=a/d/+%22%27%3Ba%26b%26cd+%60%7C+'
-#send CLAIMURL-SUCCESS
 #recv 
 
 ### ['git', '-c', 'receive.autogc=0', '-c', 'gc.auto=0', 'annex', 'find', '--debug', '--json', '--not', '--in', 'here', 'simple.txt']
 
 ### ['git', '-c', 'receive.autogc=0', '-c', 'gc.auto=0', 'annex', 'get', '--debug', '--json', '--json-progress', 'simple.txt']
 #send VERSION 1
 #recv PREPARE
 #send PREPARE-SUCCESS
 #recv TRANSFER RETRIEVE SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.txt .git/annex/tmp/SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.txt
 #send GETURLS SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.txt dl+archive:
-#recv VALUE dl+archive:SHA256E-s172--70cf6dd95738e5d3672a7139a2785b0a979f0f7955d0f6da0d94cc03c84a63b7.tar.gz#path=a/d/+%22%27%3Ba%26b%26cd+%60%7C+
+#recv VALUE dl+archive:SHA256E-s173--db0a9680f8d15578de8e4a5b5c1e87f36d9372d6118fb24c1c60f390e71ad3c1.tar.gz#path=a/d/+%22%27%3Ba%26b%26cd+%60%7C+
 #recv VALUE 
 #send TRANSFER-SUCCESS RETRIEVE SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.txt
 #recv 
 
-### ['git', '-c', 'receive.autogc=0', '-c', 'gc.auto=0', 'annex', 'rmurl', '--debug', 'simple.txt', 'dl+archive:SHA256E-s172--70cf6dd95738e5d3672a7139a2785b0a979f0f7955d0f6da0d94cc03c84a63b7.tar.gz#path=a/d/+%22%27%3Ba%26b%26cd+%60%7C+']
+### ['git', '-c', 'receive.autogc=0', '-c', 'gc.auto=0', 'annex', 'rmurl', '--debug', 'simple.txt', 'dl+archive:SHA256E-s173--db0a9680f8d15578de8e4a5b5c1e87f36d9372d6118fb24c1c60f390e71ad3c1.tar.gz#path=a/d/+%22%27%3Ba%26b%26cd+%60%7C+']
+#send VERSION 1
+#recv PREPARE
+#send PREPARE-SUCCESS
+#recv CLAIMURL dl+archive:SHA256E-s173--db0a9680f8d15578de8e4a5b5c1e87f36d9372d6118fb24c1c60f390e71ad3c1.tar.gz#path=a/d/+%22%27%3Ba%26b%26cd+%60%7C+
+#send DEBUG Claiming url 'dl+archive:SHA256E-s173--db0a9680f8d15578de8e4a5b5c1e87f36d9372d6118fb24c1c60f390e71ad3c1.tar.gz#path=a/d/+%22%27%3Ba%26b%26cd+%60%7C+'
+#send CLAIMURL-SUCCESS
+#recv 
+
+### ['git', '-c', 'receive.autogc=0', '-c', 'gc.auto=0', 'annex', 'drop', '--debug', '--json', 'simple.txt']
 send VERSION 1
 recv PREPARE
 send PREPARE-SUCCESS
-recv CLAIMURL dl+archive:SHA256E-s172--70cf6dd95738e5d3672a7139a2785b0a979f0f7955d0f6da0d94cc03c84a63b7.tar.gz#path=a/d/+%22%27%3Ba%26b%26cd+%60%7C+
-send DEBUG Claiming url 'dl+archive:SHA256E-s172--70cf6dd95738e5d3672a7139a2785b0a979f0f7955d0f6da0d94cc03c84a63b7.tar.gz#path=a/d/+%22%27%3Ba%26b%26cd+%60%7C+'
-send CLAIMURL-SUCCESS
+recv CHECKPRESENT SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.txt
+send GETURLS SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.txt dl+archive:
+recv VALUE dl+archive:SHA256E-s173--db0a9680f8d15578de8e4a5b5c1e87f36d9372d6118fb24c1c60f390e71ad3c1.tar.gz#path=a/d/+%22%27%3Ba%26b%26cd+%60%7C+
+recv VALUE 
+send CHECKPRESENT-SUCCESS SHA256E-s3--a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3.txt
 recv 
-
-### ['git', '-c', 'receive.autogc=0', '-c', 'gc.auto=0', 'annex', 'drop', '--debug', '--json', 'simple.txt']

our test verifies that annex refuses to drop the content if we remove the dl+archive link for the key, and now it fails

Comment by yarikoptic — Mon Dec 11 14:40:26 2017

Remove comment

comment 7

The diff shows git annex whereis used to send CLAIMURL to the external, and no longer does.

Reproduction recipe using git-annex-remote-ipfs:

git annex initremote ipfs type=external externaltype=ipfs encryption=none
date > somefile
git annex add somefile
git annex addurl --debug --relaxed ipfs:dummy --file somefile

Which results in git annex whereis somefile saying it's present in ipfs, but not listing the ipfs url for it. And again, whereis does not sent CLAIMURL.

And, in log.web, I see why:

+1513013502.312530881s 1 ipfs:dummy

That is not an OtherDownloader url, it's lacking the ":" prefix.

This seems to be particular to the addurl --relaxed --file code path; letting addurl add a new file does result in an OtherDownloader url being recorded.

I guess the reason the test suite then fails is, the url it removes is not the one git-annex recorded, and so git-annex still thinks it's at the wrongly recorded url, and so dropping succeeds.