Advertise grpcs:// to agents by default and accept grpcs for a KAS subdomain

Vishal Takrequested to merge
vtak/kas-grpcs-default into master

What does this MR do?

Makes native gRPC the default agent connection for Linux package installations, part of Phase 4 of &23445 (closed) (gitlab#628216 (closed)). No NGINX change is needed: the /gitlab.agent.* gRPC location has existed on the GitLab host since 18.7 (!8857 (merged)) and on the KAS subdomain vhost. This MR only changes what the cookbook advertises and accepts.

  • Derived gitlab_rails['gitlab_kas_external_url']: grpcs://<host>[:port] when external_url is HTTPS without a relative URL root and the bundled NGINX with HTTP/2 is enabled. Otherwise unchanged: wss:// or ws:// under /-/kubernetes-agent/. An explicit value still wins.
  • gitlab_kas_external_url accepts grpcs:// for a KAS subdomain and no longer requires gitlab_kas['listen_websocket'] for it. grpc:// stays rejected because NGINX serves HTTP/2 on the TLS listener only. The derived k8s proxy URL uses https for grpcs too.
  • Fixes a latent bug: the derived URL dropped the port number for non-standard ports (host: instead of host:8443).
  • gitlab.rb.template comments updated.

Existing agents connected over WebSocket are unaffected; KAS serves both protocols on one port. After upgrading, the agent installation instructions in the UI show the grpcs:// address. Installs behind a proxy that terminates TLS and talks HTTP/1.1 to NGINX need to keep advertising WebSocket with gitlab_rails['gitlab_kas_external_url']; see the 19.5 upgrade note in gitlab!254411 (merged).

Companion MRs: Helm chart gitlab-org/charts/gitlab!5333 (merged), docs gitlab!254411 (merged) (must ship in the same milestone).

How can this change be tested?

  1. Reconfigure with an HTTPS external_url and check gitlab_kas.external_url in /var/opt/gitlab/gitlab-rails/etc/gitlab.yml is grpcs://<host>.
  2. Run an agent against it: agentk --kas-address=grpcs://<host> --token-file=<file> connects. An existing agent on wss://<host>/-/kubernetes-agent/ still connects.
  3. Set gitlab_kas_external_url 'grpcs://kas.<host>/' and reconfigure: no error, KAS vhost rendered with http2 on and the gRPC location.

ChefSpec: gitlab-kas_spec.rb and nginx_spec.rb, 177 examples, 0 failures, with new examples for the default, non-standard port, HTTP/2 off, NGINX off, and the subdomain schemes.

Scripts:

kas-grpc-e2e.sh

Logs

 ./kas-grpc-e2e.sh run    

== 0. Inputs ==
MR library: vtak/kas-grpcs-default:files/gitlab-cookbooks/gitlab-kas/libraries/gitlab_kas.rb (7e53a585)
GitLab image: gitlab/gitlab-ee:nightly, agentk image: registry.gitlab.com/gitlab-org/cluster-integration/gitlab-agent/agentk:v19.3.1, host: gitlab.check:8443, KAS subdomain: kas.gitlab.check

== 1. Self-signed certificate for gitlab.check and kas.gitlab.check ==
OK   certificate written to /Users/vtak/Desktop/kas-grpc-check/kas-grpc-check/ssl, trusted copy in /Users/vtak/Desktop/kas-grpc-check/kas-grpc-check/trusted-certs

== 2. Start GitLab (stock cookbook) and wait for it ==
waiting for readiness (usually 4-6 min) ............
KAS: kas version v19.4.0-rc2, git ref: a0a94e6094e0997e4bde40b58aa2d7065d026003
stock cookbook md5: b39ad450

== 2b. Register an agent and start k3s (both agent checks need them) ==
Successfully copied 2.56kB to kas-check-gitlab:/tmp/make_agent.rb
OK   agent id 1 registered in project root/kas-grpc-check
OK   k3s ready

== 3a. Derived KAS settings with the STOCK cookbook (expect wss://, and the port-drop bug) ==
    enabled: true
    external_url: wss://gitlab.check:/-/kubernetes-agent/
    external_k8s_proxy_url: https://gitlab.check:8443/-/kubernetes-agent/k8s-proxy/
-- routing already exists with the stock cookbook (since 18.7), only the advertised URL changes:
OK   GitLab host, stock cookbook: HTTP/2 gRPC answered by KAS (grpc-message: Request unauthenticated with bearer)
OK   GitLab host, stock cookbook: WebSocket upgrade 101

== 3b. Drop in the MR library and reconfigure (~1-2 min) ==
Successfully copied 13.3kB to kas-check-gitlab:/opt/gitlab/embedded/cookbooks/gitlab-kas/libraries/gitlab_kas.rb
    enabled: true
    external_url: grpcs://gitlab.check:8443
    external_k8s_proxy_url: https://gitlab.check:8443/-/kubernetes-agent/k8s-proxy/
OK   derived external_url is grpcs://gitlab.check:8443
-- NGINX GitLab vhost:
35:  listen *:8443 default_server ssl;
38:  server_name gitlab.check;
45:  http2 on;
124:  location = /-/kubernetes-agent/ {
129:  location /gitlab.agent. {
130:    grpc_pass grpc://localhost:8150;
134:  location /-/kubernetes-agent/k8s-proxy/ {
-- an existing agent on the OLD default URL must still connect right after the change:
waiting for Rails behind NGINX ..... ok
waiting for the KAS tracker entry  ok
connected agentk instances: 1
  pod=agentk-ws version=v19.3.1
OK   agentk-ws connected over wss://gitlab.check:8443/-/kubernetes-agent/ after the change (warnings: 0)

== 3c. Probes through NGINX on the GitLab host ==
OK   GitLab host: HTTP/2 gRPC answered by KAS (grpc-message: Request unauthenticated with bearer)
OK   GitLab host: WebSocket upgrade 101

== 3d. Real agents over grpcs:// and over the old wss:// URL at the same time (two agentk containers) ==
waiting for Rails behind NGINX  ok
waiting for the KAS tracker entry  ok
-- KAS tracker keys in Redis:
gitlab-kas:agent_tracker2:conn_by_agent_id:1
-- Rails-side lookup (KAS internal API), takes ~1 min:
connected agentk instances: 2
  pod=agentk-ws version=v19.3.1
  pod=agentk-grpc version=v19.3.1
OK   agentk-grpc is connected over grpcs://gitlab.check:8443
OK   agentk-ws is connected over wss://gitlab.check:8443/-/kubernetes-agent/ (old default keeps working)
-- NGINX access log: gRPC methods (HTTP/2) and WebSocket upgrades (101) from the agents:
      1 /gitlab.agent.agent_registrar.rpc.AgentRegistrar/Register 200 HTTP/2.0"
      1 /-/kubernetes-agent/ 101 HTTP/1.1"
  1 x /-/kubernetes-agent/ HTTP/1.1 101
-- agentk warnings/errors: grpc=0 ws=0

== 3e. WebSocket fallback: HTTP/2 off on the GitLab vhost (two reconfigures, ~3 min) ==
    enabled: true
    external_url: wss://gitlab.check:8443/-/kubernetes-agent/
    external_k8s_proxy_url: https://gitlab.check:8443/-/kubernetes-agent/k8s-proxy/
OK   derived external_url falls back to wss://gitlab.check:8443/-/kubernetes-agent/ with the port intact
OK   GitLab vhost no longer has http2 on
gRPC probe with HTTP/2 off (informational): HTTP/2 200  grpc-status: 16
OK   GitLab host, HTTP/2 off: WebSocket upgrade 101
OK   HTTP/2 restored, derived external_url is grpcs://gitlab.check:8443 again
waiting for Rails behind NGINX  ok
OK   GitLab host, HTTP/2 restored: HTTP/2 gRPC answered by KAS (grpc-message: Request unauthenticated with bearer)

== 4. KAS on its own subdomain with grpcs:// (rejected by the stock cookbook) ==
    enabled: true
    external_url: grpcs://kas.gitlab.check/
    external_k8s_proxy_url: https://kas.gitlab.check/k8s-proxy/
OK   external_url is grpcs://kas.gitlab.check/
OK   k8s proxy URL derived as https
-- NGINX KAS vhost:
14:  listen *:443 ssl;
15:  server_name kas.gitlab.check;
21:  http2 on;
80:  location /gitlab.agent. {
81:    grpc_pass grpc://localhost:8150;
OK   KAS subdomain: HTTP/2 gRPC answered by KAS (grpc-message: Request unauthenticated with bearer)
OK   KAS subdomain: WebSocket upgrade 101
OK   GitLab host (still routed): HTTP/2 gRPC answered by KAS (grpc-message: Request unauthenticated with bearer)

== Done. Everything passed. Logs in /Users/vtak/Desktop/kas-grpc-check/kas-grpc-check. Run './kas-grpc-e2e.sh cleanup' to remove the containers. ==

Related issues

Related to gitlab#628216 (closed)

Checklist

See Definition of done.

For anything in this list which will not be completed, please provide a reason in the MR discussion.

Required

  • MR title and description are up to date, accurate, and descriptive.
  • MR targeting the appropriate branch.
  • Latest Merge Result pipeline is green.
  • When ready for review, MR is labeled workflowready for review per the Distribution MR workflow.
  • The UBT version and corresponding checksum hash have been updated and referenced in the merge request if applicable.
    • UBT EE pipeline (Trigger:ee-package-ubt) is green

For GitLab team members

If you don't have access to this, the reviewer should trigger these jobs for you during the review process.

  • The manual Trigger:ee-package jobs have a green pipeline running against latest commit.
  • If config/software or config/patches directories are changed, make sure the build-package-on-all-os job within the Trigger:ee-package downstream pipeline succeeded.
  • If you are changing anything SSL related, then the Trigger:package:fips manual job within the Trigger:ee-package downstream pipeline must succeed.
  • If CI configuration is changed, the branch must be pushed to dev.gitlab.org to confirm regular branch builds aren't broken.

Expected (please provide an explanation if not completing)

  • Test plan indicating conditions for success has been posted and passes.
  • Documentation created/updated.
  • Tests added.
  • Integration tests added to GitLab QA.
  • Equivalent MR/issue for the GitLab Chart opened.
  • Validate potential values for new configuration settings. Formats such as integer 10, duration 10s, URI scheme://user:passwd@host:port may require quotation or other special handling when rendered in a template and written to a configuration file.
Edited by Vishal Tak

Merge request reports