Recent comments posted to this site:

In order to use rclone as a special remote, the user needs to download a separate Bash scriptfrom https://github.com/DanielDent/git-annex-remote-rclone and put it in their PATH. Since that extra dependency is only a few hundred lines of Bash, I would be interested in attempting to implement Remote/Rclone.hs so that the rclone special remote is entirely built into git-annex. However, I wanted to run it by you before more seriously considering investing time in doing that. What are your thoughts on this? I'm assuming the only reason rclone support isn't built into git-annex is just a lack of time and incentive, rather than a more fundamental technical reason. Is that right?

Thanks for all your work on this tool.

Comment by oadams Thu Mar 14 01:03:46 2024

https://hackage.haskell.org/package/botan-low is another possibility. There is a significant effort ongoing to build up this library stack in haskell. It does need the botan C library to be installed separately unfortunately (I've suggested they embed it).

I have not benchmarked it yet but the docs say it supports hardware accellerated SHA1, SHA2, SHA3 on x86 and also SHA1, SHA2 on arm64.

Comment by joey Mon Mar 11 13:43:19 2024

From some more research it seems that Extended Master Secret (aka Extended Main Secret) is a TLS 1.2 only extension, to work around a problem with TLS 1.2 (eg, 2015 post about the problem).

TLS v1.3 doesn't have this problem, by design, AFAIK. And thus clients/servers supporting TLS v1.3 entirely avoids the problem (possibly why I have only found it on a few servers; the one I looked into in detail definitely won't connect with TLS v1.3 right now, but they're looking into it).

The webserver support can be confirmed with, eg forced TLS v1.2:

echo "" | openssl s_client -tls1_2 -connect WEBSERVER:443 2>&1 | egrep "Protocol|Extended master"

and forced TLS v1.3 to check if that will work:

echo "" | openssl s_client -tls1_3 -connect WEBSERVER:443

Hopefully that means the number of impacted sites is relatively small (eg, ones that haven't enabled TLS v1.3 support in the last 5+ years).

Ewen

Comment by ewen Thu Mar 7 03:01:20 2024

I'm not sure I understand what you mean.

Do you mean that I should clone the s3 bucket with a tool like git-remote-s3? I'm not sure how to do that.

I tried downloading the whole bucket from backblaze, but since I enabled encryption on the bucket, backblaze doesn't let me download the encrypted files. I'm not sure yet if I can get those files using their api.

Just in case I didn't explain correctly what I'm trying to do, I'm trying to do something like this, on a new computer with a fresh ~/Document:

❯ cd ~/Documents
❯ git init
Initialized empty Git repository in /home/bbigras/Documents/.git/
❯ git annex init
init  ok
(recording state in git...)
❯ git annex initremote backblaze type=S3 signature=v4 host=s3.us-west-000.backblazeb2.com bucket=my-bucket protocol=https encryption=hybrid keyid=my-key-id
initremote backblaze (encryption setup) (to gpg keys: my-key-id) (checking bucket...)
  The bucket already exists, and its annex-uuid file indicates it is used by a different special remote.

git-annex: Cannot reuse this bucket.
failed
initremote: 1 failed
Comment by bbigras Wed Mar 6 17:03:50 2024

annex.tune.objecthash1=true and annex.tune.branchhash1=true seem like they could be helpful in reducing git annex's inode usage, but the disclaimer about this feature being experimental is a little worrying.

Since this it is over 10 years old though, is it still considered experimental or has it graduated to being a stable feature? I.e. will using this meaningfully increase the chance of losing data?

Also, what is the (potential) benefit of using lowercase for the hashes?

Comment by imlew Wed Mar 6 12:26:56 2024

Went far enough down implementing registerurl --move-from to be sure that it would complicate the code far more than just adding a new moveurl command. So despite it being a fairly unusual situation, a new command is better than that option.

And implemented it:

joey@darkstar:~/tmp/x>git-annex registerurl WORM--bar http://example.com/bar.fooregisterurl http://example.com/bar.foo ok
joey@darkstar:~/tmp/x> git-annex whereis --key WORM--bar
whereis WORM--bar (1 copy)
    dca0b5f9-659a-4928-84db-ff9fd74d8fc8 -- [foo]

  foo: http://example.com/bar.foo
ok
joey@darkstar:~/tmp/x>git-annex reregisterurl WORM--bar --move-from=foo
reregisterurl WORM--bar ok
joey@darkstar:~/tmp/x>git-annex whereis --key WORM--bar
whereis WORM--bar (2 copies)
    00000000-0000-0000-0000-000000000001 -- web
    dca0b5f9-659a-4928-84db-ff9fd74d8fc8 -- [foo]

  web: http://example.com/bar.foo
ok
Comment by joey Tue Mar 5 18:11:32 2024

Thing is that unregisterurl does not mark content as not present in a special remote. Except for the web which is a special case. Reason is that not having an url registered by a special remote does not prevent getting content from that special remote in general.

So my idea for --move-from=foo was that it should behave the same way since it's the same as registerurl + unregisterurl.

If you're going to remove/disable the special remote anyway, it won't matter whether git-annex thinks it contains content, I suppose? Or you could use setpresentkey of course.

Comment by joey Tue Mar 5 17:23:34 2024

I agree with you. Specifically, this is related to git, not git-annex.

I wanted to run git annex assist just and expect to be fixed.

I wrote a simple python script to fix the issue, which changed the path of the submodule. And by adding the script in pre-commit hook, I can do git annex assist now without fixing the submodules manually. (I used python to manipulate the text files such as dot-gitmodules etc. Perl or Ruby would be much compact in this job.)

It seems to work well for my needs, even though just a prototype script.

After some polishing (and testing), I'll upload the script here.

Thank you.

Comment by TTTTAAAx Tue Mar 5 05:21:59 2024

You need to clone to git repository where you initialized the special remote the first time, then git-annex enableremote will enable using it from the clone.

Comment by joey Mon Mar 4 21:17:26 2024

Cool! Do let us know when it's available and we can add it to the list of special remotes.

WRT the simple export interface, there is a PR on git-annex-remote-rclone that adds it https://github.com/git-annex-remote-rclone/git-annex-remote-rclone/pull/62. But, it got hung up on a question of avoiding possible data loss if two git-annexes are exporting the same file. That sounds like a problem you will need to consider in your implementation too.

Comment by joey Mon Mar 4 21:10:44 2024