## Summary 1. Proposal: our runner managers should reach Relay over the internal network. Today they go out through the public internet. 2. Finding: the runner already creates the correlation ID, and it survives on both internal paths. Only the public path replaces it. ```mermaid flowchart LR subgraph ci["gitlab-ci (GCP project)"] RM["runner manager"] VM["ephemeral VM"] end subgraph prod["gitlab-production (GCP project)"] GW["CI gateway<br/>HAProxy, 10.216.14.x:8989"] API["API<br/>ingress-nginx → Workhorse → Rails"] KAS["Relay / KAS job_router<br/>internal LB 10.216.8.102:8150<br/>kas.int.gprd.gitlab.net"] end CF["Cloudflare Spectrum<br/>kas.gitlab.com:443 → 8150"] HAP["HAProxy main<br/>kas_grpc backend"] RM -->|"direct: POST /api/v4/jobs/request"| GW GW --> API RM -->|"today: grpcs://kas.gitlab.com from discovery"| CF CF --> HAP HAP --> KAS KAS -->|"POST /api/v4/internal/ci/job_router/jobs/request<br/>via internal gateway"| API RM -. "proposed: grpcs://kas.int.gprd.gitlab.net:8150" .-> KAS RM --> VM linkStyle 0,1 stroke:#e03131,stroke-width:2px linkStyle 2,3,4 stroke:#f08c00,stroke-width:2px linkStyle 6 stroke:#2f9e44,stroke-width:2px ``` Red: direct polling. Orange: Job Router today. Green: the proposal. ## 1. Internal Relay address Discovery returns `grpcs://kas.gitlab.com`. Managers in `gitlab-ci` go out to the internet, through Cloudflare Spectrum and HAProxy main, and back in to the KAS internal load balancer `10.216.8.102:8150`. That load balancer already has a name, `kas.int.gprd.gitlab.net`. Direct polling already stays internal, through the CI gateway. Proposal: dial `grpcs://kas.int.gprd.gitlab.net:8150` directly. Needed: - Runner: a config setting for the Relay address, for example `[runners.job_router] url`. Today it only uses discovery ([`router/client.go`](https://gitlab.com/gitlab-org/gitlab-runner/-/blob/main/router/client.go)). - TLS: the certificate on port 8150 must cover the internal name. To check. - Network: a firewall rule from the manager subnets to `10.216.8.102:8150`. To check. Alternatives: discovery returns an internal URL for CI gateway traffic, or split DNS for `kas.gitlab.com` plus a 443 → 8150 mapping. Going through the CI gateway instead is not worth it: it needs a new gRPC front end, and the rate limit bypass does not carry over from Relay to Rails anyway ([runbook](https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/kas/job-router.md#rate-limiting)). Rollout is per shard in runner config. Fallback to direct polling is unchanged. ## 2. Correlation ID (checked 2026-09-30) - The runner creates the ID, 32 hex characters, on both paths: `X-Request-Id` header on `jobs/request`, `x-request-id` metadata on `GetJob`. - Relay forwards it to Rails ([`server.go`](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/blob/master/internal/module/job_router/server/server.go)). Workhorse keeps it: propagation is on and `10.0.0.0/8` is trusted. nginx and HAProxy leave it alone. - On `gitlab.com/*` the Cells HTTP router Worker overwrites it with `<cf-ray>-<colo>` ([`upstream.ts`](https://gitlab.com/gitlab-org/cells/http-router/-/blob/main/src/rules/upstream.ts)). `kas.gitlab.com` is Spectrum, TCP only, so it cannot. ```mermaid flowchart LR R["runner mints X-Request-Id<br/>32 lowercase hex"] subgraph int["internal paths: ID preserved"] direction LR GW["CI gateway<br/>no header changes"] --> NG["ingress-nginx<br/>passes X-Request-ID through"] K["Relay<br/>metadata → X-Request-Id header"] --> IG["internal gateway<br/>no header changes"] --> NG NG --> WH["Workhorse<br/>propagateCorrelationID, trusts 10.0.0.0/8"] --> RL["Rails logs the runner's ID"] end subgraph pub["public path: ID replaced"] direction LR W["Cells HTTP router Worker<br/>sets X-Request-Id = cf-ray-COLO"] --> HM["HAProxy main"] --> NG2["ingress-nginx"] --> WH2["Workhorse"] --> RL2["Rails logs the ray ID"] end R -->|"POST jobs/request through the CI gateway"| GW R -->|"GetJob metadata through Spectrum and HAProxy,<br/>both leave headers alone"| K R -->|"POST jobs/request to gitlab.com<br/>customer runners, Chef shards on https://gitlab.com"| W ``` Kibana, gprd, 10 minutes: | Route | Rows | ID | Example | | --- | --- | --- | --- | | `jobs/request` via CI gateway | 4,586 | runner, 32 hex | `73799c4f15434df5896017306a6c051e` ([Discover](https://log.gprd.gitlab.net/app/kibana#/discover?_a=(columns:!(json.correlation_id,json.remote_ip,json.ua,json.status)))) | | `jobs/request` via Cloudflare, customer runners | 362,713 | ray ID | `a432c5986d2e2bdc-IAD` ([Discover](https://log.gprd.gitlab.net/app/kibana#/discover?_a=(columns:!(json.correlation_id,json.remote_ip,json.ua,json.status)))) | | `internal/ci/job_router/jobs/request` via Relay | 54,801 | runner, 32 hex | `a1a8ed0069244ac1bdf4291437cd0036` ([Discover](https://log.gprd.gitlab.net/app/kibana#/discover?_a=(columns:!(json.correlation_id,json.remote_ip,json.ua,json.status)))) | Runner logs agree ([Discover](https://log.gprd.gitlab.net/app/kibana#/discover?_a=(columns:!(json.correlation_id,json.msg,json.shard,json.runner)). Workhorse-made IDs, 26 characters, only appear for private shard runners that point at dev.gitlab.org and ops.gitlab.net. So: on internal paths the ID is the runner's, nothing to change. The internal address changes the route, not the ID. On the public path the ID is Cloudflare's, and logs still line up because the runner logs the ID it gets back. Follow-ups: - [ ] Note this in the readiness logging doc and the [Job Router runbook](https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/kas/job-router.md). - [ ] Choose the runner setting or an alternative, and open the change. - [ ] Check the certificate and the firewall. Related: gitlab-com/gl-infra/production-engineering#29694, gitlab-org/cluster-integration/gitlab-agent#943, [readiness docs](https://gitlab.com/gitlab-org/architecture/readiness/-/tree/main/features/job_router).