### Background In https://gitlab.com/gitlab-org/gitlab/-/issues/487062+ we enabled gRPC connections to GitLab Relay (KAS) on GitLab.com. HAProxy now routes HTTP/2 to a dedicated `kas_grpc` backend and HTTP/1.1 (WebSocket) to the existing `kas` backend ([frontends/kas.erb](https://gitlab.com/gitlab-cookbooks/gitlab-haproxy/-/blob/b0c61c475beb1f86226c9b98f7274c0b175e8367/templates/default/frontends/kas.erb#L36), [backends/kas_grpc.erb](https://gitlab.com/gitlab-cookbooks/gitlab-haproxy/-/blob/b0c61c475beb1f86226c9b98f7274c0b175e8367/templates/default/backends/kas_grpc.erb)). The rollout completed on 2026-08-31. Rails still advertises `wss://kas.gitlab.com` as the KAS external URL. Nothing on GitLab.com sets `externalUrl` explicitly; the GitLab chart derives it from `global.hosts.kas.name` plus HTTPS, which always produces `wss://` ([globals.md](https://gitlab.com/gitlab-org/charts/gitlab/-/blob/34e426c262dd714a7f92a06dc41d868d526aec77/doc/charts/globals.md)). Every client already supports `grpcs://`, so the remaining work is configuration, defaults and docs, plus a decision on how to move agents that are already running. ### Goal - New agents, runners, runner controllers, workspaces, and `glab` bootstraps on GitLab.com connect to KAS over `grpcs://kas.gitlab.com`. - A measurable, low-risk path for existing `wss` agents to move to gRPC. - Omnibus and the GitLab Helm chart enable gRPC and advertise `grpcs://` by default for new installs. - `wss` keeps working throughout. Deprecating it is out of scope here. ### What already handles `grpcs` No code changes are needed to flip the URL: - Rails: [`Gitlab::Kas.external_url`](https://gitlab.com/gitlab-org/gitlab/-/blob/6889b13819da1ac5273fc77eac0aecae45f5dfa4/lib/gitlab/kas.rb#L71) is passed through as-is; the scheme rewrite to `https` for the k8s proxy URL ([kas.rb#L81](https://gitlab.com/gitlab-org/gitlab/-/blob/6889b13819da1ac5273fc77eac0aecae45f5dfa4/lib/gitlab/kas.rb#L81)) and the Workspaces OAuth app ([oauth_application_attributes_generator.rb#L23](https://gitlab.com/gitlab-org/gitlab/-/blob/6889b13819da1ac5273fc77eac0aecae45f5dfa4/ee/lib/remote_development/workspaces_server_operations/server_config/oauth_application_attributes_generator.rb#L23)) accept both `grpcs` and `wss`. - agentk / agentw: [options.go#L145](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/blob/94cc2ae19bb40695138923aaee0aa5c2bbea4185/internal/cmd/agent/options.go#L145) selects gRPC TLS for `grpcs`. - Runner Job Router: [client_conn_factory.go#L217](https://gitlab.com/gitlab-org/gitlab-runner/-/blob/290853b0d8cb62c9e9e1ef5ed0585f442604a294/router/client_conn_factory.go#L217). ### Consumers of the advertised URL These pick up the new value automatically once Rails is reconfigured: | Consumer | Where | Effect of the flip | |----------|-------|--------------------| | Agent install command in the UI | [clusters_helper.rb#L29](https://gitlab.com/gitlab-org/gitlab/-/blob/6889b13819da1ac5273fc77eac0aecae45f5dfa4/app/helpers/clusters_helper.rb#L29), [cluster_agents_helper.rb#L11](https://gitlab.com/gitlab-org/gitlab/-/blob/6889b13819da1ac5273fc77eac0aecae45f5dfa4/app/helpers/projects/cluster_agents_helper.rb#L11), [clusters_util.js#L15](https://gitlab.com/gitlab-org/gitlab/-/blob/6889b13819da1ac5273fc77eac0aecae45f5dfa4/app/assets/javascripts/clusters_list/clusters_util.js#L15) | New installs get `--set config.kasAddress=grpcs://kas.gitlab.com` | | Cert-based cluster migration | [install_agent_service.rb#L142](https://gitlab.com/gitlab-org/gitlab/-/blob/6889b13819da1ac5273fc77eac0aecae45f5dfa4/app/services/clusters/migration/install_agent_service.rb#L142) | Same | | REST `GET /api/v4/metadata`, GraphQL `metadata.kas.externalUrl` | [metadata.rb#L10](https://gitlab.com/gitlab-org/gitlab/-/blob/6889b13819da1ac5273fc77eac0aecae45f5dfa4/lib/api/entities/metadata.rb#L10), [kas_metadata.rb#L14](https://gitlab.com/gitlab-org/gitlab/-/blob/6889b13819da1ac5273fc77eac0aecae45f5dfa4/app/models/app_config/kas_metadata.rb#L14) | Advertises `grpcs` | | `glab cluster agent bootstrap` | [api_wrapper.go#L266](https://gitlab.com/gitlab-org/cli/-/blob/b05b9ae251e0af83489ed75180753bb00fe537c8/internal/commands/cluster/agent/bootstrap/api_wrapper.go#L266) reads the metadata API | New bootstraps use `grpcs`, no CLI change | | Runner Job Router discovery | [runner.rb#L190](https://gitlab.com/gitlab-org/gitlab/-/blob/6889b13819da1ac5273fc77eac0aecae45f5dfa4/lib/api/ci/runner.rb#L190); runners cache for 1h ([client.go#L34](https://gitlab.com/gitlab-org/gitlab-runner/-/blob/290853b0d8cb62c9e9e1ef5ed0585f442604a294/router/client.go#L34)) | Whole runner fleet moves to gRPC within an hour. Needs monitoring. | | Workspaces (agentw) | [workspace_variables_builder.rb#L178](https://gitlab.com/gitlab-org/gitlab/-/blob/6889b13819da1ac5273fc77eac0aecae45f5dfa4/ee/lib/remote_development/workspace_operations/create/workspace_variables_builder.rb#L178) | New workspaces use `grpcs`; existing ones keep `wss` | ### Plan **Phase 1: dogfood** - [x] Add a Grafana panel splitting HAProxy `kas` vs `kas_grpc` backend sessions. This is the migration progress metric. - [x] Switch GitLab's own agents to `grpcs`: [gstg values.yaml#L4](https://gitlab.com/gitlab-com/gl-infra/argocd/apps/-/blob/5ed036c82a61f97c0791b71fc4f620b465dda9b4/services/gitlab-agent/env/gstg/values.yaml#L4), [pre values.yaml#L4](https://gitlab.com/gitlab-com/gl-infra/argocd/apps/-/blob/5ed036c82a61f97c0791b71fc4f620b465dda9b4/services/gitlab-agent/env/pre/values.yaml#L4) **Phase 2: flip the advertised URL (gstg, pre, then gprd via change request)** - [x] k8s-workloads: set `global.appConfig.gitlab_kas.externalUrl: grpcs://kas.<env>` in [gstg.yaml.gotmpl](https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/blob/4a1d7e131f6f0e797b2754e6e25e9cbbd995bc3c/releases/gitlab/values/gstg.yaml.gotmpl), [pre.yaml.gotmpl](https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/blob/4a1d7e131f6f0e797b2754e6e25e9cbbd995bc3c/releases/gitlab/values/pre.yaml.gotmpl), [gprd.yaml.gotmpl](https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/blob/4a1d7e131f6f0e797b2754e6e25e9cbbd995bc3c/releases/gitlab/values/gprd.yaml.gotmpl) - [x] chef-repo: `gitlab_kas_external_url` in [gstg-base.json#L882](https://gitlab.com/gitlab-com/gl-infra/chef-repo/-/blob/d60226758c422416118866ce6bd6d98458155b50/roles/gstg-base.json#L882), [pre-base.json#L246](https://gitlab.com/gitlab-com/gl-infra/chef-repo/-/blob/d60226758c422416118866ce6bd6d98458155b50/roles/pre-base.json#L246), [gprd-base.json#L1081](https://gitlab.com/gitlab-com/gl-infra/chef-repo/-/blob/d60226758c422416118866ce6bd6d98458155b50/roles/gprd-base.json#L1081) - [x] Verify `GET /api/v4/metadata` and the UI install command on each environment - [x] Watch runner Job Router error rates and the `kas_grpc` backend for the hour after each flip **Phase 3: defaults and docs** - [x] **Scope**: only GitLab.com references change here. Self-managed examples move with Phase 4. - [x] `charts/gitlab-agent` default `config.kasAddress`: [values.yaml#L72](https://gitlab.com/gitlab-org/charts/gitlab-agent/-/blob/1fa265adab337bf9883d9c27d257661fd30571e4/values.yaml#L72) and README - [x] Update gitlab-agent kpt package: [base.yaml#L38](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/blob/94cc2ae19bb40695138923aaee0aa5c2bbea4185/build/deployment/gitlab-agent/base.yaml#L38), [kpt-setter-configmap.yaml#L9](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/blob/94cc2ae19bb40695138923aaee0aa5c2bbea4185/build/deployment/gitlab-agent/kpt-setter-configmap.yaml#L9), [README.md#L22](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/blob/94cc2ae19bb40695138923aaee0aa5c2bbea4185/build/deployment/gitlab-agent/README.md#L22) - [x] GitLab docs: [kas.md#L25](https://gitlab.com/gitlab-org/gitlab/-/blob/6889b13819da1ac5273fc77eac0aecae45f5dfa4/doc/administration/clusters/kas.md#L25), [agent/install/\_index.md#L28](https://gitlab.com/gitlab-org/gitlab/-/blob/6889b13819da1ac5273fc77eac0aecae45f5dfa4/doc/user/clusters/agent/install/_index.md#L28), [agent/troubleshooting.md#L57](https://gitlab.com/gitlab-org/gitlab/-/blob/6889b13819da1ac5273fc77eac0aecae45f5dfa4/doc/user/clusters/agent/troubleshooting.md#L57), [set_up_gitlab_agent_and_proxies.md#L90](https://gitlab.com/gitlab-org/gitlab/-/blob/6889b13819da1ac5273fc77eac0aecae45f5dfa4/doc/user/workspace/set_up_gitlab_agent_and_proxies.md#L90), [set_up_infrastructure.md#L203](https://gitlab.com/gitlab-org/gitlab/-/blob/6889b13819da1ac5273fc77eac0aecae45f5dfa4/doc/user/workspace/set_up_infrastructure.md#L203), [workspaces_troubleshooting.md#L97](https://gitlab.com/gitlab-org/gitlab/-/blob/6889b13819da1ac5273fc77eac0aecae45f5dfa4/doc/user/workspace/workspaces_troubleshooting.md#L97) - [x] Handbook Cells diagram: [infrastructure/\_index.md#L276](https://gitlab.com/gitlab-com/content-sites/handbook/-/blob/f225d6c5bbc31bcb544833f0f5675be05e223f93/content/handbook/engineering/architecture/design-documents/cells/infrastructure/_index.md#L276) - [x] Ensure the changelog in the helm chart announces `grpcs://kas.gitlab.com` **Phase 4: gRPC by default in Omnibus and the GitLab Helm chart** **Decision**: both distributions enable gRPC routing and advertise `grpcs://` by default for new installs. `wss` stays supported. Owned by Distribution, can run in parallel with Phases 2 and 3. The advertised URL must only change in the same release that enables the routing. - [x] GitLab Helm chart: default `gitlab.kas.ingress.grpc.enabled` to `true` and derive `global.appConfig.gitlab_kas.externalUrl` as `grpcs://kas.<domain>` when it is on. Keep `wss` when a relative URL root is set. See [https://gitlab.com/gitlab-org/charts/gitlab/-/blob/master/charts/gitlab/charts/kas/templates/ingress-grpc.yaml](https://gitlab.com/gitlab-org/charts/gitlab/-/blob/34e426c262dd714a7f92a06dc41d868d526aec77/charts/gitlab/charts/kas/templates/ingress-grpc.yaml) and [https://gitlab.com/gitlab-org/charts/gitlab/-/blob/master/charts/gitlab/charts/kas/values.yaml](https://gitlab.com/gitlab-org/charts/gitlab/-/blob/34e426c262dd714a7f92a06dc41d868d526aec77/charts/gitlab/charts/kas/values.yaml) - [x] Omnibus, KAS on its own subdomain: accept `grpcs://` in `parse_gitlab_kas_external_url_using_own_subdomain` and drop the `listen_websocket` requirement. The NGINX template already routes `/gitlab.agent.*` via `grpc_pass`. See [https://gitlab.com/gitlab-org/omnibus-gitlab/-/blob/master/files/gitlab-cookbooks/gitlab-kas/libraries/gitlab_kas.rb](https://gitlab.com/gitlab-org/omnibus-gitlab/-/blob/7f1ce34b91be18046c91769322666b839ba6fa87/files/gitlab-cookbooks/gitlab-kas/libraries/gitlab_kas.rb) and [https://gitlab.com/gitlab-org/omnibus-gitlab/-/blob/master/files/gitlab-cookbooks/gitlab-kas/templates/default/nginx-gitlab-kas.conf.erb](https://gitlab.com/gitlab-org/omnibus-gitlab/-/blob/7f1ce34b91be18046c91769322666b839ba6fa87/files/gitlab-cookbooks/gitlab-kas/templates/default/nginx-gitlab-kas.conf.erb) - [x] Omnibus, default layout (KAS under `/-/kubernetes-agent/` on the GitLab hostname): add a `/gitlab.agent.` gRPC location to the main NGINX server block, and change `build_default_gitlab_kas_external_url` to emit `grpcs://gitlab.example.com` with no path when HTTPS is on. Keep `ws`/`wss` for HTTP-only and relative URL root installs. - [x] Docs: self-managed examples in [https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/administration/clusters/kas.md](https://gitlab.com/gitlab-org/gitlab/-/blob/6889b13819da1ac5273fc77eac0aecae45f5dfa4/doc/administration/clusters/kas.md), [https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/user/clusters/agent/install/\_index.md](https://gitlab.com/gitlab-org/gitlab/-/blob/6889b13819da1ac5273fc77eac0aecae45f5dfa4/doc/user/clusters/agent/install/_index.md) and [https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/user/workspace/set_up_gitlab_agent_and_proxies.md](https://gitlab.com/gitlab-org/gitlab/-/blob/6889b13819da1ac5273fc77eac0aecae45f5dfa4/doc/user/workspace/set_up_gitlab_agent_and_proxies.md), plus the chart globals and `gitlab.rb.template` comments - [x] Changelog and/or upgrade note for administrators **Phase 5: existing connections (organic migration)** No Relay client can be moved from the server side today: agentk, agentw and Runner Controllers read their KAS address from local configuration at startup, and no RPC response carries an address. We will not build a migration mechanism. Instead we rely on organic upgrades and let the data decide whether `wss` ever needs to be deprecated. How each client moves to `grpcs`: - Runner (Job Router): automatically, within an hour of the Phase 2 flip, via discovery. - agentw (Workspaces): new workspaces get the new address. Furthermore, this is not yet released to end users. - agentk: when users re-run the install command, bootstrap a new agent with `glab`, or upgrade the chart without an explicit `config.kasAddress`. - Runner Controllers: confirm how the address is configured and document the change path. Still an Experiment. ### Out of scope - ops instance ([gitlab-ops values.yaml#L201](https://gitlab.com/gitlab-com/gl-infra/argocd/apps/-/blob/5ed036c82a61f97c0791b71fc4f620b465dda9b4/services/gitlab-ops/values.yaml#L201)) unless its HAProxy has the same gRPC routing - GitLab Dedicated and self-managed defaults - Removing the WebSocket endpoint ### Related - https://gitlab.com/gitlab-org/gitlab/-/issues/487062+ - https://gitlab.com/gitlab-org/gitlab/-/issues/626738+ - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/21134+ (staging gRPC enablement) - https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/work_items/816 (production gRPC enablement) - https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/work_items/803+