Extend grpc-proxy library to enable multiple backends

In order to support routing of RPC write requests to one or more backends, the grpc-proxy library will require modification. Currently, the library only supports a single downstream server per active RPC. See the stream director interface to understand the limitation.

Some other functionality needed to accomplish our HA Beta goals:

• Inspect RPC calls and determine if the accessor or mutable StreamDirector are needed -> #1496 (closed)
• New stream director type that supports more than one downstream
• Ability to stream to downstream nodes at different rates for a given mutable RPC
• Ability to timeout/cancel a downstream node without impacting other nodes in an RPC

added 1 deleted label

/cc @zj

Take a look at the outstanding pull request activity on the current project: https://github.com/mwitkow/grpc-proxy/pulls

I think we will need to fork the project and maintain our own for velocity sake. It doesn't appear that mwitkow is actively maintaining the project anymore. The most active fork seems to belong to this person: https://github.com/iwittkau/grpc-proxy

Here's another issue we will need to figure out to enable the mutator/accessor switch logic: gitaly-proto!228 (diffs)

The diff shows that the only change in the Go generated code was the file descriptor. Our solution will need to parse the file descriptor to obtain the RPC category (accessor/mutator).

@pokstad1 By forking, do you mean a separate project? Or a package? Its a very minor question, but just curious.

RegisterNode() will only allow 1 backend right now when !1068 (merged) is merged. That should probably change before closing this issue.

@zj Yes, a separate project. In case we want to upstream our contributions, it would be helpful to keep it in a dedicated repo.

changed milestone to %11.10

added Deliverable label

Ability to timeout/cancel a downstream node without impacting other nodes in an RPC

This should be a separate issue, @zj to split out

Inspect RPC calls and determine if the accessor or mutable StreamDirector are needed

Should be its own issue, not needed right. But should be in the beta.

• Ability to stream to downstream nodes at different rates for a given mutable RPC

We should split this out into a separate issue. This will likely involve a buffer of some type to allow different rates. It will be somewhat complex.

Inspect RPC calls and determine if the accessor or mutable StreamDirector are needed

Should be part of #1496 (closed)

changed milestone to %11.11

unassigned @pokstad1

changed the description

mentioned in merge request !1147 (merged)

@jramsay @zj If we aren't worried about transactions for the beta, then this can be moved out to the top level epic. I will make a replacement ticket for extending/rewriting the proxy to inspect traffic.

removed from epic &289 (closed)

removed milestone