Spike: Investigate Cloudflare gRPC support

For Duo Workflow support, the current design doc suggests an Executor component that talks to a Runway service via gRPC. This service would be proxied through cloud.gitlab.com as any other CC backend.

We should investigate Cloudflare support for gRPC services. The docs mention that for fully proxied DNS records, which is what we use, it should be supported: https://developers.cloudflare.com/network/grpc-connections/

However, some things to explore or verify:

• "Charges may occur for gRPC traffic over add-on products such as Argo Smart Routing, WAF, and Bot Management." (we use WAF -- what are these extra charges?)
  • Outcome: We confirmed there won't be additional charges: "Hi Matthias -- Confirmed that if you use gRPC with any of your contracted services (for example, WAF) and traffic, there are no additional charges. GitLab isn't currently contracted for Bot Management and Argo, so adding these services would be an additional cost. However, using gRPC with these services, after they're purchased, does not incur additional fees. Hope that helps to clarify. Thanks!"
• Investigate support for long-running connections (see also https://grpc.io/docs/guides/keepalive/#common-situations-where-keepalives-can-be-useful)
  • Outcome: This seems to work. We confirmed we were able to hold a connection open for ~50 minutes.
• gRPC support needs to be enabled for cloud.gitlab.com; it is currently disabled. We should:
  • Update TF config to enable it for cloud.staging.gitlab.com
  • Consider writing a small POC that exposes a gRPC service, or alternatively already integrate with a POC for the Workflow engine, if it exists
  • Outcome: The cloudflare-terraform plugin does not support this gRPC setting. We ended up flipping this switch manually in the Cloudflare web UI. We had to make changes to our transform rules that drop the path prefix, because this was breaking RPCs. Otherwise we found it works fine.