simplify praefect routing to primary and replication nodes
simplify praefect routing to no longer track primary and replicas per repository. This overly complicates things and we only really need to know which replicas are were, which we already have through the replication jobs.
Before this MR, this is what praefect's stream director did.
1. First Request for {"storage_name":"default", "relative_path":"ab/de/deadbeef"} comes to praefect
- praefect gets an RPC request for a repository
{"storage_name":"default", "relative_path":"ab/de/deadbeef"} - praefect looks in the database to see if
ab/de/deadbeefis already tracked - it is not tracked.
- praefect takes the default primary,
internal-gitaly-0and secondariesinternal-gitaly-1,internal-gitaly-2, and writes a record to the datastore that tracks which primary and which secondariesab/de/deadbeefgets written to. - praefect rewrites storage
defaulttointernal-gitaly-0 - praefect creates replication jobs
- praefect proxies the request over to
internal-gitaly-0
2. Subsequent Requests for {"storage_name":"default", "relative_path":"ab/de/deadbeef"} comes to praefect
- praefect gets an RPC request for a repository
{"storage_name":"default", "relative_path":"ab/de/deadbeef"} - praefect looks in the database to see if
ab/de/deadbeefis already tracked - it is tracked.
- praefect finds a record like
{"relative_path":"/ab/de/deadbeef", "primary":"internal-gitaly-0", "secondaries":["internal-gitaly-1", "internal-gitaly-2"]}. Thus, it knows that it will need to proxy the request tointernal-gitaly-0 - praefect creates replication jobs
- praefect proxies the request over to
internal-gitaly-0
This design is overly complex because there is no need for praefect to track per repository which node praefect will proxy the request to, and which secondaries it will create replication jobs for. I think at one point we wanted the flexibility to use any node as a primary, and we wanted the granularity to specify which repositories go to which primaries. But, as we have it now, this intricacy is unnecessary.
All praefect needs to know is "which node is the primary for this virtual storage" and "which nodes are the secondaries for this virtual storage".
Also, we already create replication jobs, we already have a record of which repositories have been replicated to which secondaries.
This means we can get rid of the code that tracks which repository is going to which node in the database.
After this MR, this is what praefect's stream director will do
1. First Request for {"storage_name":"default", "relative_path":"ab/de/deadbeef"} comes to praefect
- praefect gets an RPC request for a repository
{"storage_name":"default", "relative_path":"ab/de/deadbeef"} - praefect looks for the virtual storage
default, and which primary is configured for it. The primary isinternal-gitaly-0 - praefect rewrites storage
defaulttointernal-gitaly-0 - praefect creates replication jobs
- praefect proxies the request over to
internal-gitaly-0