bug: BackendTrafficPolicy breaks KAS Kubernetes API proxy (Port 8154) due to forced h2c protocol error

Description

When migrating to the Kubernetes Gateway API using Envoy Gateway, enabling global.kas.enabled generates a BackendTrafficPolicy that applies useClientProtocol: true across the entire gitlab-kas HTTPRoute.

While this correctly handles multiplexed gRPC traffic for cluster agents on Port 8150, it simultaneously forces standard HTTP/1.1 REST traffic destined for the KAS Kubernetes API proxy (Port 8154) to be forwarded as cleartext HTTP/2 (h2c). Because Port 8154 does not support cleartext HTTP/2, kubectl requests (like logs, exec, or proxy requests) immediately fail with an Envoy protocol reset error.

Steps to Reproduce

  1. Deploy the GitLab Helm chart with Gateway API routing enabled (ingress-nginx disabled).
  2. Configure KAS with internal TLS disabled (global.kas.tls.enabled: false) and the proxy enabled (global.kas.k8sProxy.enabled: true).
  3. The chart generates a single HTTPRoute named gitlab-kas containing rules for both the root path / (port 8150) and /k8s-proxy/ (port 8154).
  4. The chart generates a BackendTrafficPolicy targeting the entire gitlab-kas HTTPRoute with useClientProtocol: true.
  5. Attempt a kubectl command targeting the KAS proxy.

Expected Behavior

kubectl commands successfully tunnel through the KAS proxy to the cluster.

Actual Behavior

The client receives a protocol reset from Envoy:

err="couldn't get current server API group list: an error on the server ("upstream connect error or disconnect/reset before headers. reset reason: protocol error")

Technical Root Cause

The BackendTrafficPolicy targets the entire HTTPRoute resource by name:

spec:
  targetRefs:
  - group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: gitlab-kas
  useClientProtocol: true

Because both port 8150 (gRPC) and port 8154 (HTTP/1.1 REST) are defined within the same HTTPRoute, Envoy forces useClientProtocol: true on both backends. When a user runs a kubectl command using an HTTP/2 connection to Envoy, Envoy attempts to speak h2c to port 8154, causing KAS to drop the connection.

Proposed Fix

The Helm chart should isolate these routing concerns by splitting them into two separate HTTPRoute resources when the Kubernetes Gateway API provider is active:

  1. gitlab-kas (HTTPRoute): Handles the root path / for agent traffic on port 8150. The BackendTrafficPolicy should exclusively target this route.
  2. gitlab-kas-k8s-proxy (HTTPRoute): A dedicated route handling /k8s-proxy/ pointing to port 8154. This route must not be targeted by the BackendTrafficPolicy, allowing Envoy to gracefully downgrade HTTP/2 client connections to HTTP/1.1 before reaching the KAS proxy container.

Versions

  • Chart: 10.1.1
  • Platform:
    • Cloud: GKE
  • Kubernetes:
    • Client: v1.35.6-dispatcher
    • Server: v1.35.5-gke.1057002
  • Helm:
    • Client: v4.2.2
    • Server: ?

Relevant logs

kubectl get pods
E0625 12:17:07.357252 1210349 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: an error on the server (\"upstream connect error or disconnect/reset before headers. reset reason: protocol error\") has prevented the request from succeeding"
E0625 12:17:07.388330 1210349 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: an error on the server (\"upstream connect error or disconnect/reset before headers. reset reason: protocol error\") has prevented the request from succeeding"
E0625 12:17:07.421384 1210349 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: an error on the server (\"upstream connect error or disconnect/reset before headers. reset reason: protocol error\") has prevented the request from succeeding"
E0625 12:17:07.455070 1210349 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: an error on the server (\"upstream connect error or disconnect/reset before headers. reset reason: protocol error\") has prevented the request from succeeding"
E0625 12:17:07.487188 1210349 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: an error on the server (\"upstream connect error or disconnect/reset before headers. reset reason: protocol error\") has prevented the request from succeeding"
Error from server (InternalError): an error on the server ("upstream connect error or disconnect/reset before headers. reset reason: protocol error") has prevented the request from succeeding