Connecting to a discussion we had at distribits....
It would be useful to extend the external special remote protocol with the ability to create ephemeral special remotes. Ephemeral in the sense that they are created by and during the runtime of a special remote, and only exist until that special remote process is terminated by git-annex.
There could be a new protocol command that takes the same parameters as initremote as arguments. Its response would be the UUID of the created special remote.
The second part of the protocol extension would be a third response value for CHECKPRESENT, TRANSFER*, REMOVE. The addition to SUCCESS, and FAILURE would by REDIRECT-REMOTE <UUID>, and instruct git-annex to perform the same request against the special remote given by UUID instead.
The corresponding change in key availability would be recorded for the original special remote.
A use case would be to have an "orchestration" special remotes that maybe represent a particular infrastructure. They dynamically deploy appropriate transfer setups, and do not commit them to a repository. This can be useful for setups with short-lived tokens/urls. This is
in some way also an alternative to the sameas approach, where the alternatives are hidden in the implementation of a special remote, rather than in each repository.
The major difficulty in implementing this seems to be the setup stage, which is the per-special-remote code that runs during initremote/enableremote. That code can write to disk, or perform expensive operations.
A few examples:
My gut feeling is that it won't be practical to make it possible to ephemeralize every type of special remote. But it would not be too hard to make some subset of special remotes able to be used ephemerally.
It might be possible to maintain a cache of recently used ephemeral special remotes across runs of git-annex, and so avoid needing to re-run the setup stage.
There are also some common setup stage tasks that pose problems but could all be fixed in one place: