Combine listenmux and streamrpc into a publicly exposed test RPC

For gitlab-com/gl-infra/scalability#1151 (closed)

Problem

In !3601 (merged), we already implement the library code for StreamRPC protocol. The protocol works well as a standalone protocol on top of gRPC over TCP. This MR continues to integrate the StreamRPC protocol into Gitaly, and expose a public RPC called TestStream. As its name already suggested, the RPC is to test the integration, as a prerequisite for the next real implementation of the new Git fetch protocol.

TestStream RPC is just a ping-pong RPC that validates input repository, and echos the first size bytes of the incoming stream back to the client.

service TestStreamService {
  rpc TestStream(TestStreamRequest) returns (google.protobuf.Empty) {
    option (op_type) = {
op: ACCESSOR
    };
  }
}

message TestStreamRequest {
  Repository repository = 1 [(target_repository)=true];
  int64 size = 2;
}

The outcome is to confirm the following criteria:

• StreamRPC works seamlessly with Gitaly gRPC server. It means the client can call StreamRPC call to Gitaly binding gRPC address. StreamRPC must be able to handle different types of wires:
  • Insecure TCP
  • TLS over TCP
  • Unix socket
• StreamRPC should be able to re-use existing gRPC interceptors, including, but not limited to:
  • Logging
  • Prometheus metrics
  • Authentication

TestStream RPC is a new gRPC call. It is even not registered with Gitaly server. Hence, the change should not affect the existing main activities.

Implementation details

The full flow with TLS looks like following:

sequenceDiagram
  Client->>Gitaly Server: TCP Handshake
  Gitaly Server->>Client: TCP Handshake
  Gitaly Server-->>Gitaly Listenmux: Delegate
  Client->>Gitaly Listenmux: TLS Handshake
  Gitaly Listenmux->>Client: TLS Handshake
  Client->>Gitaly Listenmux: Write "streamrpc00" magic bytes
  Note over Gitaly Listenmux: Pick either<br/>backchannel or<br/>streamrpc or <br/>normal flow
  Gitaly Listenmux-->>StreamRPC Server: Spawn a goroutine
  Gitaly Listenmux-->>Gitaly Server: Return 
  Gitaly Server->>Gitaly Server: Skip without error
  Note over Client: Method:<br/>"/gitaly.TestStreamService/TestStream"<br/>Metadata:<br/>{"authorization": "Bearer abc"}<br/>Message:<br/>&TestStreamRequest{Size: 1024}
  Client->>StreamRPC Server: StreamRPC Handshake: [4-byte length][JSON Handshake Payload]
  Note over StreamRPC Server:Verify and Accept
  StreamRPC Server->>Client: StreamRPC Handshake: \x00\x00\x00\x00
  StreamRPC Server-->>TestStream Handler:Invoke with<br/>TestStreamRequest{Size: 1024}
  Client->>TestStream Handler: Byte stream
  TestStream Handler->>Client: Byte stream
  Note over TestStream Handler, Client: Close connection after done

To accomplish this flow, there are some significant changes:

• Update Gitaly server factory to create streamrpc servers, corresponding to each Gitaly gRPC server. The streamRPC server shares unary server interceptor with gRPC server.
• Implement streamrpc.Handshaker, and register streamrpc to listenmux multiplexer instance in each Gitaly server. The handshaker is similar to backchannel Handshaker. It acts as a wrapper to invoke stream RPC server to handle the request.
• Add teststream.proto, consisting of TestStreamService and TestStream stream RPC call.
• Other changes are mostly new tests, adding stream RPC support to existing test setup, etc.

Manual testing

Call over Insecure TCP, Unix Socket, and TLS without authentication

Call over Insecure TCP, Unix Socket, and TLS with authentication

Prometheus metrics