## Summary A customer with **"Restrict access by IP address"** enabled on their top-level group reports inconsistent behavior between two runner-manager fleets serving the same `saas-linux-small-amd64` tag: - Jobs dispatched to `5-green.saas-linux-small-amd64.runners-manager.gitlab.com` (**Chef** fleet) → succeed. - Jobs dispatched to `k8s.saas-linux-small-amd64.runners-manager.gitlab.com` (**k8s** fleet) → fail (403 / unable to reach GitLab API). The customer's group allowlist contains only their own office/VPN CIDRs, so hosted-runner traffic is expected to be exempt via GitLab's **globally-allowed IP address ranges** (the [Global allow list](https://docs.gitlab.com/administration/settings/visibility_and_access_controls/#configure-globally-allowed-ip-address-ranges)). The k8s fleet's traffic is being treated as coming from an unrecognized IP. This is a **Chef→k8s migration gap** (relates to &29, which moves the runner-manager pods onto GKE). ## Root cause Both fleets' **ephemeral job VMs** live on the same Shared-VPC subnets in `gitlab-ci-155816` and their CIDRs are already covered by the global allowlist (`10.14.0.0/16` covers `saas-l-s-amd64` p1–p9 = `10.14.64.0/21` … `10.14.128.0/21`). So the ephemeral VMs are **not** the problem. The difference is the **runner-manager** itself: - **Chef managers** run on VMs in the `gitlab-ci/ci/runner-managers` subnet `10.1.5.0/24` — which **is** in the global allowlist. - **k8s managers** run as **pods on the GKE cluster**, so manager→GitLab API calls originate from the **GKE node/pod ranges**, which are **NOT** in the global allowlist. For the `saas-l-s-amd64` cluster (`gke-runners-saas-l-s-amd64`), per the [ephemeral-runners CIDR SSOT](https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/ci-runners/ci-runner-networking.md#ephemeral-runners-unique-cidrs-list): | Purpose | CIDR | |---|---| | GKE nodes | `10.53.0.0/20` | | GKE pods | `10.54.0.0/16` | | GKE services | `10.55.0.0/20` | None of these are in the current global allowlist. That is the divergence that makes the k8s fleet fail where the Chef fleet succeeds. ## Current global allowlist (for reference) `10.222.0.0/16, 10.64.0.0/16, 10.65.0.0/16, 10.66.0.0/16, 10.67.0.0/16, 10.68.0.0/16, 10.1.5.0/24, 10.10.0.0/16, 10.11.0.0/16, 10.12.0.0/16, 10.13.0.0/16, 10.14.0.0/16` Mapping of the runner-relevant entries: - `10.1.5.0/24` — `gitlab-ci/ci/runner-managers` (Chef manager VMs) - `10.10.0.0/16` — `saas-l-m-amd64-gpu-s` + first block of `r-saas-l-s-amd64` ephemeral subnets - `10.11.0.0/16` — GPU shard tail - `10.12.0.0/16` — `saas-l-p-amd64` (private) + `saas-l-s-arm64` - `10.13.0.0/16` — `saas-l-m-amd64`, `saas-l-l-amd64`, `saas-l-xl-amd64`, `saas-l-2xl-amd64`, `saas-l-m-arm64`, most of `saas-l-l-arm64` - `10.14.0.0/16` — `saas-l-l-arm64` tail, `saas-l-m-amd64-org`, **`saas-l-s-amd64` ephemeral VMs (p1–p9)** - `10.64.0.0/16`–`10.68.0.0/16`, `10.222.0.0/16` — GitLab internal infra ranges (not runner projects) ## Proposed fix Add the GKE manager cluster ranges for the migrated k8s fleet to the [global allowlist](https://docs.gitlab.com/administration/settings/visibility_and_access_controls/#configure-globally-allowed-ip-address-ranges) (Admin → Settings → General → Visibility and access controls): - [ ] `10.53.0.0/20` (nodes — `gke-runners-saas-l-s-amd64`) - [ ] `10.54.0.0/16` (pods) - [ ] `10.55.0.0/20` (services) And, going forward, as &29 centralizes managers onto `runner-managers-gprd-1` in `gitlab-ci`, add **that** cluster's node/pod ranges too (`saas-l-p-amd64/gke-runners`: `10.50.0.0/20`, `10.51.0.0/16`, `10.52.0.0/20`, plus the runner-2/3 secondary ranges) so all k8s-hosted managers are exempt. ## Follow-ups - [ ] Confirm which traffic (manager API/polling vs. job artifact/web redirect) actually egresses from the GKE pod range vs. the ephemeral VMs, to validate the exact set of ranges required. - [ ] Make "add the manager GKE node/pod/service CIDRs to the global allowlist" an explicit step in the k8s runner-manager **bring-up checklist** (runner-managers runbook), analogous to the existing ephemeral-CIDR step in [scale-existing-shards.md](https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/ci-runners/linux/scale-existing-shards.md#document-cidrs). Missing the ephemeral step previously caused prod#18952; this is the manager-side equivalent. - [ ] Reconcile the `r-saas-l-s-amd64` p7–p9 subnet values (`10.14.136.0/21`+ in Terraform) with the SSOT table. ## References - Epic: gitlab-com/gl-infra/ci-runners&29 (Centralize runner managers on one GKE cluster) - CIDR SSOT: https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/ci-runners/ci-runner-networking.md#ephemeral-runners-unique-cidrs-list - Global allow list docs: https://docs.gitlab.com/administration/settings/visibility_and_access_controls/#configure-globally-allowed-ip-address-ranges - Prior related incident (ephemeral CIDR missed): gitlab-com/gl-infra/production#18952