Hi folks, I'm working on a new external special remote for rclone named "git-annex-remote-rclone-goyle", or "garrgoyle" for short. (I chose a unique name to disambiguate from the preexisting git-annex-remote-rclone project.)
Like git-annex-remote-rclone, garrgoyle only supports the external special remote protocol. In the near future, I'd like to add support for the simple export interface (see export_and_import_appendix).
Garrgoyle seems to be faster than git-annex-remote-rclone. Anecdotally, it was 10x faster when I asked to copy ~20 files that were already present on a "drive" remote. I think the difference is that garrgoyle pays the rclone startup cost once, rather than once per action.
I've uploaded a PR to rclone's GitHub repo https://github.com/rclone/rclone/pull/7654. If anyone is willing to alpha test, I would greatly appreciate it! Feedback on the PR is also welcome.
Cool! Do let us know when it's available and we can add it to the list of special remotes.
WRT the simple export interface, there is a PR on git-annex-remote-rclone that adds it https://github.com/git-annex-remote-rclone/git-annex-remote-rclone/pull/62. But, it got hung up on a question of avoiding possible data loss if two git-annexes are exporting the same file. That sounds like a problem you will need to consider in your implementation too.
This has now been merged into rclone, but it's not yet shipped in a release. It is now a subcommand of rclone (
rclone gitannex) rather than a standalone command. When invoked with-h, it prints some rough installation instructions, which are also available here: https://github.com/rclone/rclone/blob/master/cmd/gitannex/gitannex.md.As for the concurrent transfers problem, I was told by rclone's maintainer that on backends where partially uploaded files are visible, the backend automatically writes to a temp location and renames when the transfer is complete [GitHub thread]. I want to delve into this a little more and do something in
rclone gitannexwhen the backend does not have this two-step-transfer property (maybe it should print a warning, or maybe just abort).See external special remotes not using git-annex-remote in name for possible changes in git-annex related to this.
Looking at the docs at https://github.com/git-annex-remote-rclone/git-annex-remote-rclone there are a lot of options for different directory layouts. Do you have any plans to support those? It seems it might be useful for migration of existing special remotes, if nothing else.
I guess that "rcloneprefix=" in this is the same as "prefix=" in git-annex-remote-rclone?
I've added a mention of this to rclone.
This wasn't on my roadmap, but I think it's worth prioritizing. It would enable users to migrate between these special remote implementations without requiring double the storage space on the remote and a proportional amount of patience. I've added this to milestone 2 in my tracking comment on GitHub.
Correct. I intentionally chose to start all config names with "rclone". I think this makes it a little easier for users to grok the initremote command, since other key-value pairs like
encryption=hybridalso show up there. I suppose I could also support config names without the rclone prefix without much trouble, and it might make it easier for users to migrate. I'll have to mull on that. Also added to the tracking comment linked above.Awesome! Thanks for adding my project! Two edits:
rclone gitannex -h. Without-h, it will start speaking the external special remote protocol at the user