Goal: cut the cost and operational overhead of running one manager GKE cluster per ephemeral-VM project. Today that is private-runner-1/2/3 and s-l-s-amd64-runner-1, with six more implied by the private shard's remaining projects. And the private shard is the small case: production has more than 50 ephemeral VM projects across the shards. Finishing the migration on this architecture means 50-plus clusters, each paying the GKE management fee (about $73 per month, so roughly $45k per year in fees alone) and each carrying its own minimum node footprint, monitoring stack, network, shared subnet, PSC endpoints, Workload Identity bindings, vault entries, and ArgoCD registration. The larger cost is operational: setting up a cluster takes eight steps across three repositories, every change that affects all clusters multiplies by their count, and four clusters in we have already been bitten twice by copied files drifting from their source. This epic replaces that with one manager cluster per environment. It builds the production one: `runner-managers-gprd-1` in the gitlab-ci project, on the network the Chef managers use today. Ops, dev, and staging runners get their own environments' clusters when their migrations come up. We name it `-1` so that adding `runner-managers-gprd-2` later stays cheap, whether for growth or because the only way to rebuild a GKE cluster is to build a new one next to it. ## 🗺️ Before and after ```mermaid graph LR subgraph before ["Before: one manager cluster per project"] direction LR subgraph bp1 ["gitlab-r-saas-l-p-amd64-1"] direction TB bc1["GKE private-runner-1<br/>manager pods"] --> bv1["job VMs"] end subgraph bp2 ["gitlab-r-saas-l-p-amd64-2"] direction TB bc2["GKE private-runner-2<br/>manager pods"] --> bv2["job VMs"] end subgraph bp3 ["gitlab-r-saas-l-p-amd64-3"] direction TB bc3["GKE private-runner-3<br/>manager pods"] --> bv3["job VMs"] end subgraph bp4 ["gitlab-r-saas-l-s-amd64-1"] direction TB bc4["GKE s-l-s-amd64-runner-1<br/>manager pods"] --> bv4["job VMs"] end bp1 ~~~ bp2 ~~~ bp3 ~~~ bp4 end classDef mgr fill:#bbdefb,stroke:#1565c0,color:#0d2c54 classDef vm fill:#c8e6c9,stroke:#2e7d32,color:#1b3d1f class bc1,bc2,bc3,bc4 mgr class bv1,bv2,bv3,bv4 vm ``` ```mermaid graph TB subgraph after ["After: one manager cluster, VMs unchanged"] direction TB subgraph ci ["gitlab-ci"] rm1["GKE runner-managers-gprd-1<br/>all shard manager pods"] end subgraph ap1 ["gitlab-r-saas-l-p-amd64-1"] av1["job VMs"] end subgraph ap2 ["gitlab-r-saas-l-p-amd64-2"] av2["job VMs"] end subgraph ap3 ["gitlab-r-saas-l-p-amd64-3"] av3["job VMs"] end subgraph apn ["... one per project"] avn["job VMs"] end rm1 --> av1 rm1 --> av2 rm1 --> av3 rm1 --> avn end classDef mgr fill:#bbdefb,stroke:#1565c0,color:#0d2c54 classDef vm fill:#c8e6c9,stroke:#2e7d32,color:#1b3d1f class rm1 mgr class av1,av2,av3,avn vm ``` ## 🚚 The job VMs don't move The job VMs never lived on the per-project cluster networks. Both Chef and k8s managers create VMs on Shared VPC subnets in gitlab-ci-155816 (`saas-l-p-amd64-p1` and friends). The per-project networks host manager pods only. So moving the pods doesn't change job isolation at all: the ephemeral projects stay the blast-radius, quota, and IAM boundary for workloads. That was always the part of the cellular design that actually mattered. The per-project control planes were just overhead on top of it. The Chef managers already do everything this cluster needs to do, from the same network: reach ci-gateway, vault, and the GCP APIs, and SSH to VMs in all nine projects. ## 🛡️ The gVisor question The original reason for a cluster per project was gVisor: run kubernetes-executor jobs next to the managers on the same cluster. The kubernetes executor can target a remote cluster (it is an API client with a kubeconfig), so this should keep working without colocation. Validating that is deferred: the docker+machine shards this epic migrates don't touch the kubernetes executor, so the experiment (a manager on the central cluster running a real job on a separate gVisor-enabled cluster) happens when kubernetes-executor workloads come up, not before. gVisor workload clusters then become plain data-plane cells that we add without touching the control plane, the same way we add ephemeral VM projects today. Whether those are also shared clusters or one per project is open, and nothing here depends on the answer. ## 🏗️ Build 1. Cluster terraform in environments/ci as explicit resources. The gke-clusters module exists to stamp identical per-project clusters, so it stays with the legacy clusters and retires with them. Regional cluster in us-east1. Manager pools are standard-8 in five machine families so mixed shard profiles bin-pack onto shared nodes, with standard-4 system pools in three families. Pods are capped at 32 per node, so the pods /16 covers 1024 nodes. 2. Firewall: manager pods range to the VM subnets (SSH, 2376) and to ci-gateway, mirroring the Chef manager rules. 3. Workload Identity: one member for the manager KSA on the central runner-manager GSA, which already has VM-create rights in every ephemeral project. Nothing per project. 4. One vault entry, one ArgoCD registration, AppProject destinations. Monitoring needs no new files, it comes from the gprd-ci env layer. ## 🐢 Migrate, one shard at a time The k8s-to-k8s move works like the Chef migration, which we have now done enough times to trust: deploy the shard to runner-managers-gprd-1 at a limit of 1, reusing the shard's existing token (GitLab treats both clusters as the same runner, so jobs split between them), check that everything works (a real job, polling, metrics labels, logs, vector and wiz), then raise capacity here. Running both at full size is fine (the VM projects are built for double capacity during rolling deploys), and once everything an old cluster served is replaced, it gets torn down rather than wound down. A manager release binds to one ephemeral VM project through its MachineOptions, so the central cluster runs one release per shard and project. The dedicated argocd-runner-managers chart expands a projectCount in the shard's app.yaml into those releases and derives the per-project GCP fields, so adding another project to the cluster is a one-line change. The field derivation currently rides on helm templating in the runner config. If https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/7185 (environment variable expansion in machine options) merges and ships, that dependency can move into the runner itself. We migrate what we have. The saas-small capacity increase continues on s-l-s-amd64-runner-1, and that shard moves like everything else when its turn comes. Sizing a shard's runner config: - concurrent is the job capacity per pod. Multiply by replicas for the per-project capacity, and by projectCount for the shard's total on the cluster. - limit = concurrent plus slack for machines mid-replacement. For docker+machine, limit caps total machines including ones being created and removed, and jobs are only accepted when a warm machine exists. At MaxBuilds 1 every job replaces its machine, so a steady share of the budget is always mid-lifecycle: chef sizes that at about 8 percent (1200/1300). At MaxBuilds 40 churn is a fortieth of that, and IdleCount-sized slack is plenty. - request_concurrency decides the job share against other managers polling the same token, independent of capacity. Keep it proportional when splitting capacity across clusters. - IdleCount is the warm pool per pod. Copy the legacy shard's value, they were tuned per worker. ## 🧹 Decommission, per retired cluster The GKE cluster, its node pools, the gke-runners subnet and secondary ranges, the WI binding and vault entry, the ArgoCD registration, and the cluster's PSC endpoints. The PSC audit needs care: some endpoints serve the ephemeral VMs (CI JWT auth to vault), not the managers, and those stay. The per-shard networks themselves stay until the VM-side PSC question is answered. ## 💸 What goes away - We pay one management fee and run one monitoring stack and one set of nodes, instead of four today and 50-plus where the old path was heading. - Each cluster had node pools sized for its own shard's peak. One shared set of pools packs the mixed manager profiles together, and the autoscaler works with all manager pods instead of three at a time. - The six remaining private cluster setups never happen, nor the roughly 45 more implied by the other shards' projects. Changes that affect every cluster stop multiplying. - Far fewer copied files. The scrape and capture-group regressions both came from per-cluster copies drifting from their source. - The cluster naming decision stops mattering, because clusters 4 through 9 are never built.