Skip to content

Create "sidechannel" as a sub-protocol of "backchannel"

Quang-Minh Nguyen requested to merge qmnguyen0711/1216-create-sidechannel-as-a-sub-protocol-of-backchannel into master

For gitlab-com/gl-infra/scalability#1216 (closed)

The sidechannel mechanism is to open a side connection inside the hanging RPC handler in the Gitaly server that allows us transfer data directly without going through gRPC. We don't want to establish a real secondary connection over the network, because of many potential reliability issues. Yamux is the answer here. We already apply a very similar flow with "backchannel". However, we cannot use back channel because backchannel starts a gRPC server in the client side and establishes gRPC connections backward, while the very first purpose of the epic is to avoid gRPC. Recently, all the connections between Praefect and Gitaly are using backchannel. We properly don't want to establish another yamux session on top of yamux session. The new sidechannel mechanism must work well with backchannel.

One way of achieving this is to to inject sidechannel inside backchannel.

sequenceDiagram
  participant Workhorse
  participant sidechannel handshaker
  participant backchannel handshaker
  participant Gitaly
  Note over Workhorse,sidechannel handshaker: Workhorse process
  Note over Gitaly,backchannel handshaker: Gitaly process
  Workhorse->>backchannel handshaker: backchannel
  backchannel handshaker->>Gitaly: gRPC
  Gitaly->>sidechannel handshaker: sidechannel
  sidechannel handshaker->>Workhorse: yamux stream

In details:

  • Implement a sidechannel registry. This registry accounts for full life-cycle management of sidechannel connections.
  • When the client setups sidechannel, it registers a callback with the registry, and receives a Sidechannel ID. When making the gRPC request, it injects this ID into outgoing metadata. The callback will be executed when the connection arrives.
  • Implement sidechannel server-side handshaker. This handshaker is responsible for pushing the raw connection, which is a yamux stream created by backchannel, into the registry and skip gRPC handling. Add listenmux and this handshaker into the gRPC server spawn by backchannel client-side handshaker.
  • Update backchannel server-side handshaker to embed the yamux session into AuthInfo, along side with the existing backchannel stream.
  • When Gitaly RPC handler wants to open the sidechannel, it extracts yamux session out of AuthInfo, creates a new yamux stream. The magic bytes of sidechannel and sidechannel id from the client are then written into the wire. Sidechannel server-side handshaker in the client receives this connection and push to the registry.

This approach works perfectly with Praefect. We'll have to turn on backchannel conditionally in other clients (workhorse for example).

Edited by Quang-Minh Nguyen

Merge request reports