todo/publicurl config for all special remotesyohhttp://git-annex.branchable.com/todo/publicurl_config_for_all_special_remotes/git-annexikiwiki2020-09-01T20:49:59Zcomment 1http://git-annex.branchable.com/todo/publicurl_config_for_all_special_remotes/comment_1_58b2ec259bd3d9e368240e459330a738/Ilya_Shlyakhter2019-01-25T21:34:15Z2019-01-25T21:34:15Z
Maybe, it'll suffice to just register the URL with the built-in web remote (via addurl, or registerurl and setpresentkey)?
comment 2http://git-annex.branchable.com/todo/publicurl_config_for_all_special_remotes/comment_2_d9933855e8521ebed00406a6350837b7/yarikoptic2019-01-26T00:46:03Z2019-01-26T00:46:03Z
Yes, it is possible to register a url per each file as a workaround but it is somewhat unnecessary and inflexible: if top url changes, all urls will need to be adjusted.
comment 3http://git-annex.branchable.com/todo/publicurl_config_for_all_special_remotes/comment_3_42ac954b120205650a2a5f03872085c2/joey2020-06-17T01:18:32Z2020-01-29T15:04:06Z
<p>I'm not sure how to implement this in git-annex's Remote API.
retrieveKeyFile/retrieveExport would need to check it and download
the url, so that would need modifications of those methods of every
remote that implements this. And it would need to be possible to
enable the remote in readonly mode.</p>
<p>It might be possible to use a mixin to modify a Remote to support this?</p>
comment 4http://git-annex.branchable.com/todo/publicurl_config_for_all_special_remotes/comment_4_3c144c0323d49f7ca2d6b9971ad3160a/joey2020-06-17T01:18:32Z2020-01-30T15:19:55Z
<p>This will need the remote to provide a function <code>Key -> FilePath</code>,
in order to support whatever hash directories or filename mangling the
remote does. It might be better to generalize the function to
<code>Url -> Key -> Url</code> where the first url is the publicurl value.
(When exporttree=true, the function is probably not needed.)</p>
<p>To support that function in external special remotes, the protocol would
need to be extended. Hmm, that means that, in order to get a file, the
external program would need to be installed, even though the actual file
download only needs http. Contrast with the current readonly mode that
doesn't need the external program to be installed since the url is recorded
on the git-annex branch.</p>
<p>I think that the only built-in remotes that would make sense to support
this are rsync, directory[1], and webdav. s3 already supports it but could
be refactored. git remotes already support http access which is effectively
the same result, and git-lfs already supports unauthed downloads, assuming
the server allows it.</p>
<p>[1] a bit problimatic because old versions used a different
hash directory than current versions, so unless it can return two urls,
things stored with an old version won't be accessible</p>
comment 5http://git-annex.branchable.com/todo/publicurl_config_for_all_special_remotes/comment_5_1d9186506f8f44415706f3b4394c9008/joey2020-07-01T19:16:06Z2020-07-01T18:59:48Z
<p><span class="createlink"><a href="http://git-annex.branchable.com/ikiwiki.cgi?do=create&from=todo%2Fpublicurl_config_for_all_special_remotes%2Fcomment_5_1d9186506f8f44415706f3b4394c9008&page=generic_readonly_http_remote" rel="nofollow">?</a>generic readonly http remote</span> if implemented would accomplish the same
thing as this todo.</p>
<p>The advantage to that idea is, it doesn't need modifications be made to every
special remote that might end up exposed over http. As long as the special
remote is not <em>too</em> special about how it mangles keys into paths, it
can work for a lot of special remotes.</p>
<p>The directory special remote is a good example.. It would be weird for that to
have http-specific configuration and complications.</p>
<p>And, that approach doesn't complicate the external special remote protocol,
and avoids the problems discussed in comment #4.</p>
<p>So, I'm inclined to that approach over this one.</p>
comment 6http://git-annex.branchable.com/todo/publicurl_config_for_all_special_remotes/comment_6_7f027aa4b5b495a93aee3cd3417862a8/joey2020-09-01T19:19:04Z2020-09-01T19:09:30Z
<p>I've implemented the http special remote, that can be combined with other
special remotes to access them using anonymous http.</p>
<p>I think that probably addresses this todo well enough to close it.
(Although I didn't get around to
<span class="createlink"><a href="http://git-annex.branchable.com/ikiwiki.cgi?do=create&from=todo%2Fpublicurl_config_for_all_special_remotes%2Fcomment_6_7f027aa4b5b495a93aee3cd3417862a8&page=todo%2Fmake_http_special_remote_support_exporttree_remotes" rel="nofollow">?</a>make http special remote support exporttree remotes</span> yet, and this
todo mentions supporting exporttree. Should be easy to add later though.)</p>
<p>There are probably some special remotes that are unusual enough that the
http special remote can't support them, which it would make sense to add a
publicurl= config to, like S3 has. (Although I think S3 itself could now be
used with the http special remote so its option is vestigal now.)</p>
<p>I guess that publicurl= config would best be added to the individual
special remote, so it doesn't need any particular support in git-annex to
add it.</p>
comment 7http://git-annex.branchable.com/todo/publicurl_config_for_all_special_remotes/comment_7_c35df3621c7b6a1b518787fdc8967ee7/yarikoptic2020-09-01T20:49:59Z2020-09-01T20:49:59Z
Awesome, Thank you! I <a href="https://git-annex.branchable.com/todo/generic_readonly_http_remote/#comment-f6e076370814fcfab3a4d203a1adf326">still</a> think that it could be named more generically than <code>http</code> (e.g. probably could remap to public <code>rsync://</code>, <code>ftp://</code> or <code>s3://</code> to which git-annex knows how to "talk to") even if ATM supporting only <code>http/https</code>, but I guess time will show if that would be needed.