## Goal Deliver the production successor to the experimental gvisor infrastructure. The [HRPv5 experiment](https://gitlab.com/groups/gitlab-com/gl-infra/ci-runners/-/work_items/22) proved that internal CI workloads run well as gvisor-sandboxed pods on GKE, on the `gitlab-org-experimental-gvisor` cluster. This work rebuilds that stack on an architecture that can carry customer traffic and grow to the fleet's scale, then validates it against the same evaluation workloads. ## What the experiment cannot grow into The experimental cluster fails every create-time prerequisite for the 1,000 to 5,000 node tier (regional Standard, Dataplane V2, Private Service Connect: it is zonal, legacy dataplane, public), and its design predates the central-manager architecture: - Managers run in-cluster next to the jobs, one manager deployment per cluster: the per-cluster overhead the manager centralization ([epic 29](https://gitlab.com/groups/gitlab-com/gl-infra/ci-runners/-/work_items/29)) exists to remove. - Job logs ship to Cloud Logging and the observability tenant, held back only by job pods lacking `app` labels. - Deploys kill running jobs (1h grace against 4h job timeouts), image pulls trust the node cache across tenants, and the cluster mixes system, manager, and job workloads without taints. ## What this delivers - **`runner-workloads` clusters**: regional, Dataplane V2, PSC, gvisor node pools tainted and sized for job density. One cluster per ephemeral project, every cluster identically named and configured (cattle: project 2 is one terraform module block, zero apps-repo changes). Network in the gitlab-ci Shared VPC host, the layout the VM fleet already uses. - **`gprd-ci-workloads` environment**: shared by every workload cluster across shards and projects, so the cluster basics scaffold once, and its vault path space never holds manager secrets. - **Central managers, remote execution**: the runner managers stay on runner-managers-gprd-1 and drive job pods over the GKE API, authenticating with Workload Identity against an in-cluster RoleBinding, the same pattern ArgoCD uses. No static credentials. The kubeconfig is discovered at pod start, and the argocd-runner-managers chart composes the per-project wiring from the projects mechanism. - **Job logs exist only in GitLab**: Log Router exclusion, otel collector exclude list, no vector-agent in the environment. - **`p-gv-s` canary shard**: registered on the [evaluation project](https://gitlab.com/gitlab-org/production-engineering/runners-platform/gitlab) with the same tags as the experimental runner, so identical workloads compare the two stacks directly. ## Why one to two projects instead of nine The analysis that sized this (full numbers in the first notes below): the VM fleet's 9-way project split exists for per-project API churn and external IP quotas, both artifacts of the VM-per-job model. For pods, CPU quota moves with the traffic (n4 draws from a separate quota family than the VM fleet's N2D, so capacity shifts without a seesaw), and the binding constraints are storage quota and the etcd Pod-object budget, both of which land on two clusters per shard at full consolidation. Measured peaks: ~35.5k vCPU, 14.1 jobs/s, against a 100 pods/s per-cluster churn budget. ## Remaining work Status and decisions are in the notes, latest at the bottom. This list is the scope left before gvisor can take real traffic. - **Observability inside gvisor.** Job pods have no metrics today (job traces are the only log sink, nothing scrapes the `runner-workloads` namespace). Decide what per-job and per-node signal is needed and how it gets to Mimir without reopening the log path. Includes the `project` external label in k8s-monitoring-stack, the one hard prerequisite before a second cluster. - **Capacity buffers.** The pause-pod pattern or GKE capacity buffers (pre-GA) so scale-from-zero node pools do not put cold-start time into `queued_duration`. - **Concurrency and capacity.** Raise `concurrent` on the three shards from 20 / 10 / 5 (the mirror is a day behind), rerun the same-SHA comparison at that level with e2e child pipelines included, and find where the node cap of 150 and the API server bite. - **Density and efficiency.** Measure peak memory and CPU per job class against the fixed pod sizes (10Gi / 16Gi / 32Gi, request = limit) and decide whether requests can come down or be split from limits without adding OOMs. - **Opt-in path.** Which gitlab-org workloads go first, and the controls to do it safely: opt in a percentage of jobs rather than a tag, pause the shard to stop the bleeding, and let a project or job opt out. The off-gvisor list so far (`compile-production-assets`, `bundle-size-review`, `gitlab-advanced-sast`, `gitlab-advanced-sast-ext`) stays off. - **Manager identity.** The gvisor managers run on `runner-managers-gprd-1` and drive job pods with the manager's GCP identity, so https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/29709 (tighten the GCP identity behind the Kubernetes runner managers) applies here before customer jobs do. - **Later: VPA.** Run jobs as Kubernetes Jobs and let VPA size them (https://gitlab.com/gitlab-com/runner-group/team-tasks/-/work_items/437). - **Later: non-internal workloads.** Service port discovery from images (https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/6037) and whatever else customer jobs need that gitlab-org jobs do not.