Envoy Gateway: default HTTP/2 stream window (64 KiB) caps Geo SSH push proxying to ~300 KiB/s at high RTT

Summary

On a GitLab Geo deployment using Envoy Gateway (Gateway API, global.gatewayApi.installEnvoy: true), a single git push over SSH to a secondary site — which the secondary proxies to the primary over HTTP/2 — is throttled to ~300 KiB/s when the two sites are geographically distant (~200 ms RTT).

Root cause: the webservice section-scoped ClientTrafficPolicy (for the gitlab-web / gitlab-web-geo listeners) does not set the HTTP/2 flow-control windows, so Envoy Gateway applies its defaults — initialStreamWindowSize = 64 KiB, initialConnectionWindowSize = 1 MiB. A single HTTP/2 stream is bound by window / RTT, so 64 KiB over ~200 ms ≈ ~320 KiB/s, regardless of available bandwidth.

This affects any Geo-acceleration deployment on Envoy Gateway where the inter-site RTT is non-trivial (self-managed and Dedicated alike).

Reproduction

  • Two-region Geo, Envoy Gateway ingress, ~200 ms RTT between primary and secondary.
  • git push a repo with a real (incompressible) pack of tens of MB over SSH to the secondary. The secondary forwards it to the primary's gitlab-web listener over HTTP/2.
  • Observed steady-state throughput: ~300 KiB/s, flat and non-ramping (a fixed flow-control window, not congestion/BDP ramp). Parallel pushes each get the full ~300 KiB/s and scale linearly, confirming a per-stream window limit rather than aggregate bandwidth.

A direct HTTP/1.1 upload to the same ingress is fast (multi-MB/s), and an nginx-ingress Geo deployment is unaffected — isolating the cause to the HTTP/2 per-stream window on the Envoy Gateway listener.

Root-cause detail

  • The secondary→primary git-over-SSH proxy forwards as an HTTP request to the primary's gitlab-web listener; over HTTPS this negotiates HTTP/2.
  • The chart renders section-scoped ClientTrafficPolicy resources for the webservice listeners (webservice/templates/clienttrafficpolicy.yaml, clienttrafficpolicy-geo.yaml). These currently set only path.escapedSlashesAction and no http2 block.
  • Envoy Gateway gives a section-scoped policy precedence over the gateway-wide gatewayApiResources.envoy.clientTrafficPolicySpec (they do not merge), so setting http2 on the gateway-wide policy has no effect on gitlab-web. The window must be set on the section-scoped policy.
  • With the section policy leaving http2 unset, Envoy Gateway's default initialStreamWindowSize (64 KiB) applies.

Note: !5183 (merged) makes these section-scoped specs overridable by users, which is necessary — but a deployment that doesn't override them still inherits the 64 KiB default. This issue is about the default being too small for the Geo use case.

Validation

Patching the section-scoped policies (gitlab-web and gitlab-web-geo) to raise the windows:

http2:
  initialStreamWindowSize: 8Mi
  initialConnectionWindowSize: 16Mi

changed the proxied SSH push from ~300 KiB/s to ~7 MB/s (~24×) on a real high-RTT Envoy Geo deployment, with no other changes.

Proposal

Set a sensible non-default HTTP/2 window on the webservice section-scoped ClientTrafficPolicy (at least for the Geo listener gitlab-web-geo, ideally gitlab-web too), and/or document guidance for Geo-acceleration deployments.

On the suggested values (not set in stone)

The throughput ceiling of one HTTP/2 stream is initialStreamWindowSize / RTT (the bandwidth-delay product). To keep the window from being the bottleneck you want window ≥ target_throughput × RTT:

  • initialStreamWindowSize: 8Mi → at 200 ms RTT allows up to ~40 MB/s per stream (8 MiB / 0.2 s), i.e. well above typical link speeds for the Geo pair. This is the value we validated.
  • initialConnectionWindowSize: 16Mi → set to 2× the stream window so the connection-level window never becomes the limit for a single stream (Envoy's default connection window is only 1 MiB).

These are engineering estimates covering a ~150–250 ms Geo inter-region BDP, not a measured optimum — maintainers may prefer different values or a more conservative default. For reference, Envoy's own high-BDP tuning guidance is roughly 256 KiB–16 MiB, and Go's HTTP/2 client defaults its stream window to 4 MiB. Any value in the low-single-digit-MiB range and up would resolve the cap; the key point is that the current 64 KiB default is far too small for a high-RTT Geo path.

References

  • Internal investigation and full RCA: gitlab-com/gl-infra/gitlab-dedicated/team#13015
  • Related: !5183 (merged) (makes webservice section-scoped ClientTrafficPolicy specs overridable) and #6571 (closed)