This is a summary todo covering several subprojects, which would extend git-annex to be able to use proxies which sit in front of a cluster of repositories.

  1. passthrough proxy
  2. p2p protocol over http
  3. balanced preferred content
  4. track free space in repos via git-annex branch
  5. proving preferred content behavior

table of contents

planned schedule

Joey has received funding to work on this. Planned schedule of work:

  • June: git-annex proxies and clusters
  • July, part 1: p2p protocol over http
  • July, part 2: git-annex proxy support for exporttree
  • August: balanced preferred content
  • September: streaming through proxy to special remotes (especially S3)
  • October: proving behavior of balanced preferred content with proxies

work notes

  • Next step: Ready to begin implementing in servant. I have a file servant.hs in the httpproto branch that works through some of the bytestring streaming issues.

  • Perhaps: Support cgi program that proxies over to a webserver speaking the http protocol.

completed items for July's work on p2p protocol over http

  • addressed ?P2P locking connection drop safety

  • finalized HTTP P2P protocol draft 1, draft1

items deferred until later for passthrough proxy

  • Check annex.diskreserve when proxying for special remotes to avoid the proxy's disk filling up with the temporary object file cached there.

  • Resuming an interrupted download from proxied special remote makes the proxy re-download the whole content. It could instead keep some of the object files around when the client does not send SUCCESS. This would use more disk, but without streaming, proxying a special remote already needs some disk. And it could minimize to eg, the last 2 or so. The design doc has some more thoughts about this.

  • Streaming download from proxied special remotes. See design. (Planned for September)

  • When an upload to a cluster is distributed to multiple special remotes, a temporary file is written for each one, which may even happen in parallel. This is a lot of extra work and may use excess disk space. It should be possible to only write a single temp file. (With streaming this won't be an issue.)

  • Indirect uploads when proxying for special remote (to be considered). See design.

  • Getting a key from a cluster currently picks from amoung the lowest cost remotes at random. This could be smarter, eg prefer to avoid using remotes that are doing other transfers at the same time.

  • The cost of a proxied node that is accessed via an intermediate gateway is currently the same as a node accessed via the cluster gateway. To fix this, there needs to be some way to tell how many hops through gateways it takes to get to a node. Currently the only way is to guess based on number of dashes in the node name, which is not satisfying.

    Even counting hops is not very satisfying, one cluster gateway could be much more expensive to traverse than another one.

    If seriously tackling this, it might be worth making enough information available to use spanning tree protocol for routing inside clusters.

  • Optimise proxy speed. See design for ideas.

  • Use sendfile() to avoid data copying overhead when receiveBytes is being fed right into sendBytes. Library to use: https://hackage.haskell.org/package/hsyscall-0.4/docs/System-Syscall.html

  • Support using a proxy when its url is a P2P address. (Eg tor-annex remotes.)

completed items for June's work on passthrough proxy:

  • UUID discovery via git-annex branch. Add a log file listing UUIDs accessible via proxy UUIDs. It also will contain the names of the remotes that the proxy is a proxy for, from the perspective of the proxy. (done)

  • Add git-annex updateproxy command (done)

  • Remote instantiation for proxies. (done)

  • Implement git-annex-shell proxying to git remotes. (done)

  • Proxy should update location tracking information for proxied remotes, so it is available to other users who sync with it. (done)

  • Implement git-annex initcluster and git-annex updatecluster commands (done)

  • Implement cluster UUID insertation on location log load, and removal on location log store. (done)

  • Omit cluster UUIDs when constructing drop proofs, since lockcontent will always fail on a cluster. (done)

  • Don't count cluster UUID as a copy in numcopies checking etc. (done)

  • Tab complete proxied remotes and clusters in eg --from option. (done)

  • Getting a key from a cluster should proxy from one of the nodes that has it. (done)

  • Implement upload with fanout to multiple cluster nodes and reporting back additional UUIDs over P2P protocol. (done)

  • Implement cluster drops, trying to remove from all nodes, and returning which UUIDs it was dropped from. (done)

  • git-annex testremote works against proxied remote and cluster. (done)

  • Avoid git-annex sync --content etc from operating on cluster nodes by default since syncing with a cluster implicitly syncs with its nodes. (done)

  • On upload to cluster, send to nodes where its preferred content, and not to other nodes. (done)

  • Support annex.jobs for clusters. (done)

  • Add git-annex extendcluster command and extend git-annex updatecluster to support clusters with multiple gateways. (done)

  • Support proxying for a remote that is proxied by another gateway of a cluster. (done)

  • Support distributed clusters: Make a proxy for a cluster repeat protocol messages on to any remotes that have the same UUID as the cluster. Needs extension to P2P protocol to avoid cycles. (done)

  • Proxied cluster nodes should have slightly higher cost than the cluster gateway. (done)

  • Basic support for proxying special remotes. (But not exporttree=yes ones yet.) (done)

  • Tab complete remotes in all relevant commands (done)

  • Display cluster and proxy information in git-annex info (done)