Recent comments posted to this site:

comment 7

The situation with proxied remotes is:

  • --proxyconnections is the number of idle connections to cache. It is not a cap on the maximum number. If two requests come in to two different proxied remotes, a connection will be started to both, and once both are done, one of them will remain cached.
  • --proxyconnections is per-repository, so even if it were an upper bound, there could be a lot of processes for proxied external special remotes
  • Evicting a proxied connection from the pool doesn't stop the external special remote process. This is because Remote.list is still populated, and also Remote.External registers a cleanup action that stops it -- which is another reference and keeps it running until quiesce. (I have a proof of concept fix for this in p2phttpchill.)

I think that --proxyconnections can stay as an idle connections cache limit rather than an upper bound on the number of active connections. But make it be across all served repos, rather than per-repo.

It seems ok for the total number of proxy connections across all repostories to be capped by the existing limits on the things that use them. Basically that is --jobs + --lockedfiles. (With clusters, --jobs * --clusterjobs + --lockfiles)

There needs to be a Map from a proxy remote to Annex state. When opening a proxy connection, do it in the Annex state for that particular proxy remote. That way, when a proxied connection is evicted from the pool, quiesce can be used to stop it, without also stopping other proxied connections or other stuff.


Update:

In p2phttpchill I've done some work toward the above now. Implemented the proxy Map, and quiescing and GC.

There seems to be a bug left, where more than --proxyconnections proxies can be left running sometimes.

Comment by joey —
comment 6

Got this implemented, except for proxied remotes.

Leaving the todo open for that.

Comment by joey —
comment 5

Question: PerRepoServerState also has an annexState and annexRead that are not part of the AnnexWorkerPool so would not be affected by that. Are those used anywhere that spawns helper processes?

Unfortunately, it appears that they are. A git cat-file helper process is started outside the AnnexWorkerPool, so there is one per served repository.

Comment by joey —
comment 1

The flag is documented as "Build with memory rather than ram", so I don't see a problem with git-annex depending on memory as well as ram when the flag is not set. So I've made that change.

Comment by joey —
comment 4

This is a harder problem than I thought.


There is currently no way to promptly and fully stop all helper processes belonging to an AnnexState (or an AnnexRead). Two examples of processes that are not stopped by stopCoProcesses are external special remotes and external backends.

Similarly, there is no way to close every file handle that might be held open by an AnnexState. Eg, a Remote might open a Handle and keep it open.

This could be fixed, but it would be a lot of work and add rarely used code paths in multiple Remotes, etc. Which, if they're buggy, shows up as p2phttp using more resources than expected for a period of time.

Letting the AnnexState get garbage collected will stop all helper processes and close all file handles, but only once the GC runs. If new AnnexStates are being spawned faster than they're being forgotten, that could result in more helper processes and file handles being allocated than expected. So relying on GC doesn't seem feasible, unless the GC is forced to run at a particular point (System.Mem.performBlockingMajorGC). Forcing GC runs seems dodgy and might hurt performance?


When dupState is used to create a pool of AnnexStates that all cooperate on one repository, stopCoProcesses on any of them stops the helper processes of all of them. There is no way to only stop 1 of each type of helper process. So it's not possible to cool down only one of the AnnexStates that has not been recently used, while leaving the rest warmed up. This is particularly a problem for the cat-file process, which is shared amoung all AnnexStates in the pool, and so can't possibly be stopped without cooling down the whole pool. (It might be solvable for the other helper processes.)

If dupState is not used and AnnexStates are fully independent, then stopCoProcesses can be used to cool one down, but then each time one is warmed up it has to restart everything.


Here are two approaches that avoid the problems of stopCoProcesses:

  1. Bounded pool of PerRepoServerStates. When starting a new one, cool down all the AnnexStates in the pool for the LRU one. (Which stopCoProcesses will do just fine.)
  2. No dupState, so all AnnexStates are fully independent. Before using an AnnexState, if there are too many warmed up, find the LRU one and cool it down.

With #1, there would need to be another option, like --max-concurrent-repos. The maximum number of running helper processes would be on the order of that times --jobs. (Though it can be much less, eg when all traffic is going to a few repositories and other repositories are not being used.)

With #2, the total number of running helper processes would be on the order of --jobs.

The benefit of #1 is that often the same few repos will get most of the traffic. In that case, helper processes get started once and used for a long time even when the server is fully loaded.

But #1 has an edge case with very bad behavior: A fully loaded server is serving several requests for frequently used repos and 1 request for a little-used repo comes in. That request needs to block until all currently active requests for one of the active repos are complete. And for fairness, while that request is blocked, no new requests can be started that would keep an active repo active. And, once that request is running, new requests for the now inactive repo also have to block.

With #2, a fully loaded server will frequently have the LRU AnnexState getting cooled down, and then soon after warmed up and needing to restart everything. That will increase CPU load when the server is fully loaded. But it won't cause any long-term stalls, requests will always be handled in the order received.

So #2 seems better.

Comment by joey —
comment 2

Refinement: Each PerRepoServerState already has a WorkerPool of Annex objects.

When returning an Annex object to the pool, decide if it should be left warmed up, or be cooled down (stopping helper processes etc).

To decide, have a quantity semaphore. Check it when getting an Annex object from the WorkerPool. If there are too many warm objects, block until one gets cooled down.

Question: PerRepoServerState also has an annexState and annexRead that are not part of the AnnexWorkerPool so would not be affected by that. Are those used anywhere that spawns helper processes?

Comment by joey —
comment 2
d'oh -- right! installed 10.20261026+git114-ge7b75e58eb-1~ndall+1 form con/datalad builds and it works now. Thanks!
Comment by yarikoptic —
comment 1

Another approach would be to keep a bounded pool of PerRepoServerStates. When a PerRepoServerState needs to be added to the pool and the pool is full, first find one for a different repository that is not in use, and remove it from the pool, stopping its helper processes.

This seems easier to implement than coordinating a process pool between Annex objects.

And it could also avoid problems with too many open files, since it could also close open file handles in the Annex object such as the keys db handle. I do think that running out of file descriptors would be the next problem after too many processes when scaling p2phttp up to serving a large number of repositories, so this might as well solve that too.

There would still be worker pools for processes like git hash-object that are shared amoung all threads serving the same repository, scaling up to one per cpu. So long as the pool is capped at some reasonable number (say 100), that is ok, it means something like 100 * (cpus + 1) processes, probably well under the maximum number of processes. Make the maximum concurrent repositories served a tunable value.

Comment by joey —