> This epic is part of [&22 — [HRPv5] GKE Infrastructure for Runner Managers and gVisor Workloads](https://gitlab.com/groups/gitlab-com/gl-infra/ci-runners/-/epics/22). ## DRI @rehab ### Participants - @igorwwwwwwwwwwwwwwwwwwww ## Why does it matter Job pods on the `runner-workloads` clusters get one static pod-level size per shard (`pod_cpu_request` / `pod_memory_limit`: 10 / 16 / 32 GiB). Measured on `p-gv-s` ([#29646, density note](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/29646#note_3784245632)): median job uses 38% of its 10 GiB, p99 84%; requests describe ~2.5x what pods hold. The earlier dissertation measurements on `gitlab-org-experimental` showed the same shape at the job level (96.7% of jobs under half their memory, 1.4% still OOM). On Kubernetes, node count and therefore cost follow the *sum of requests*, so HRPv5's cost case ([&22](https://gitlab.com/groups/gitlab-com/gl-infra/ci-runners/-/epics/22) exit criterion, [Block 2](https://gitlab.com/groups/gitlab-com/gl-infra/ci-runners/-/epics/6)) depends on requests tracking demand. **Déjà Vu** sizes each job pod from what the same job needed last time: a mutating admission webhook sets pod-level requests/limits from the job's own history (`p95 memory x 1.2 x failure multiplier`, `mean of per-run p95 CPU`); an observer records every finished job. Jobs without history keep the shard defaults. This complements, rather than replaces, the manual per-shard request/limit tuning in [#29646](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/29646): the shard defaults remain the fallback that Déjà Vu overrides only when it has evidence. Design: [handbook!21095](https://gitlab.com/gitlab-com/content-sites/handbook/-/merge_requests/21095) (draft). Runbook: [runbooks!11623](https://gitlab.com/gitlab-com/runbooks/-/merge_requests/11623). ## Exit Criteria - [x] Déjà Vu deployed on `runner-workloads` (project 1) in **shadow mode**: every job pod carries `dejavu/cpu`, `dejavu/memory`, `dejavu/mode` annotations; observer writing `job_observations.csv` - [ ] Shadow analysis published on this epic: coverage (share of pods with history), would-OOM rate on jobs with history (target 0), projected node-hour saving vs shard defaults, gVisor sandbox memory floor, OOM classification validated on a deliberately OOM-killed sandboxed job - [ ] Duration budget agreed with DevExp (proposed: p95 job duration within 5% of shard defaults) - [ ] **Enforce** mode on the canary scope with success rate >= baseline, OOM on jobs with history = 0, duration within budget - [ ] Enforce mode on all `runner-workloads` clusters, 4 weeks stable, node-hour reduction confirmed in FinOps dashboards - [ ] Design document merged with `status: accepted` ## Details ### Shape One Pod (webhook + observer containers) per workloads cluster in namespace `runner-workloads-system`, sharing a ReadWriteOnce volume. `MutatingWebhookConfiguration` with `failurePolicy: Ignore`, scoped to `runner-workloads` and the executor's `manager.runner.gitlab.com/name` label: an unavailable webhook costs nothing but sizing. TLS from cert-manager (self-signed CA per environment). GitLab token is a Terraform-rotated `gitlab-org` group access token; GCP access is Workload Identity. Everything is Helm values in `argocd/apps`; stage changes are one-line MRs. ### Delivery (in merge order) | # | Repo | Change | MR | |---|------|--------|----| | 0 | [`gl-infra/dejavu`](https://gitlab.com/gitlab-com/gl-infra/dejavu) (new project) | Source, tests, CI building `webhook` and `observer` images | created; `main` pushed | | 0b | `gl-infra/dejavu` | Observation store moves from CSV to Prometheus: observer remote-writes one sample per finished job, webhook reads recording-rule statistics, `/metrics` | [dejavu!1](https://gitlab.com/gitlab-com/gl-infra/dejavu/-/merge_requests/1) | | 1 | `gl-infra/charts` | Chart `gitlab/dejavu` 0.1.0: stateless Deployment, dedicated `Prometheus` CR (15 d, 20 GiB), `PrometheusRule`, `PodMonitor` | [charts!1247](https://gitlab.com/gitlab-com/gl-infra/charts/-/merge_requests/1247) | | 2 | `config-mgmt` | GSA `dejavu-observer` + Workload Identity (`r-saas-l-p-amd64`); Vault auth for `runner-workloads-system` | [config-mgmt!15460](https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/merge_requests/15460) | | 3 | `argocd/config` | `runner-workloads` AppProject: allow `MutatingWebhookConfiguration/dejavu` and the External Secrets CRB; ignore the Prometheus claim as an orphan | [config!453](https://gitlab.com/gitlab-com/gl-infra/argocd/config/-/merge_requests/453) | | 4 | `argocd/apps` | cert-manager on `gprd-ci-workloads` (in-cluster CA) | [apps!3745](https://gitlab.com/gitlab-com/gl-infra/argocd/apps/-/merge_requests/3745) | | 5 | `infra-mgmt` | Register the `dejavu` project (Vault roles for semantic-release/Renovate); `gitlab-org` group access token (read_api, rotated) written to Vault `k8s/env/gprd-ci-workloads/ns/runner-workloads-system/dejavu` | [infra-mgmt!3298](https://gitlab.com/gitlab-com/gl-infra/infra-mgmt/-/merge_requests/3298) | | 6 | `argocd/apps` | `services/gitlab-runner-dejavu/` in shadow mode with the dedicated Prometheus (depends on 0b-5 and a `v1.0.0` image) | [apps!3746](https://gitlab.com/gitlab-com/gl-infra/argocd/apps/-/merge_requests/3746) | | 7 | `runbooks` | `docs/ci-runners/linux/dejavu.md` | [runbooks!11623](https://gitlab.com/gitlab-com/runbooks/-/merge_requests/11623) | | — | `handbook` | Design document | [handbook!21095](https://gitlab.com/gitlab-com/content-sites/handbook/-/merge_requests/21095) | ### Not in this iteration (tracked in the design document's open questions) CPU limit unset (currently `limit == request`); honouring author-set `KUBERNETES_*` overwrite variables; normalising parallel job names (`rspec 3/32`); rolling history window; Prometheus metrics; the RL policy and retraining loop from the dissertation. ## Work Items - [x] Delivery MRs 0-7 above (all open; merge in order, `infra-mgmt` early so the project's release jobs get their Vault roles and `v1.0.0` images can be cut) - [ ] Shadow analysis on `runner-workloads` (project 1) - [ ] Prometheus metrics and alerts (OOM-on-history, drift) before enforcing - [ ] Decide CPU-limit and overwrite-variable questions; canary - [ ] Enforce across workloads clusters - [ ] Merge the design document ## References - Design document: [handbook!21095](https://gitlab.com/gitlab-com/content-sites/handbook/-/merge_requests/21095) - Production follow-ups it complements: [#29646 Bring gVisor job execution to production](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/29646), [#28473 Properly support different sizes of runners under gVisor](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28473) - Hassanein, R. (2026). *Proactive Reinforcement Learning for Non-Interruptible CI/CD Resource Allocation in Kubernetes.* MSc dissertation, University of Liverpool.