## Problem The GitLab Operator CI runs cluster tests against **9 always-on clusters**: GKE 1.33/1.34/1.35, OpenShift 4.18/4.19/4.20/4.21, vcluster 1.35, and vcluster-flux (Envoy Gateway). Each environment requires: - A 5-job chain per pipeline (`build_review_*` → `create_review_*` → `review_*` → `qa_*` → `stop_review_*`) plus manual full-suite QA jobs - A GitLab agent connection (`gitlab-org/distribution/infrastructure/kube-agents`) - ExternalDNS + wildcard TLS certificates on `k8s-ft.win` domains - An always-on cluster that must be upgraded, right-sized, and paid for 24/7 regardless of pipeline activity Shared clusters also mean shared contention: concurrent pipelines compete for the same node pools, causing rollout timeouts and flaky failures unrelated to the code under test. Beyond resource contention, shared clusters accumulate **stale cluster-scoped state** from other pipelines. Concrete occurrence (2026-07-03, [job 15171434504](https://gitlab.com/gitlab-org/cloud-native/gitlab-operator/-/jobs/15171434504)): `review_gke133` failed on an unrelated MR because a previous `renovate/kind-0.x` pipeline left a stale `ValidatingWebhookConfiguration` on the shared GKE 1.33 cluster pointing at a webhook service whose namespace had already been torn down — every subsequent GitLab CR create on that cluster was rejected with `failed calling webhook "vgitlab.kb.io": service "gb693a6bc-renovate-kind-0-x-webhook-service" not found`, until manual cleanup. Per-job k3d clusters make this entire failure class impossible: no state outlives the job. ## Prior art: the GitLab chart migration The chart project already replaced its equivalent setup with **per-job ephemeral k3d clusters** — see gitlab-org/charts/gitlab#6421 for the full design, the sequenced MRs, and the findings recorded during implementation. Key design points that transfer directly: - [k3d](https://k3d.io/) (k3s in Docker) clusters created inside the job's DinD environment in ~30s, pinned to a K8s version via `--image rancher/k3s:<version>` - Runs on the `e2e` runner fleet (privileged DinD; `saas-linux-large-amd64` runners do **not** work) - DNS via [nip.io](https://nip.io) using the runner's Docker bridge IP (parse `ip route get 1` by the `src` keyword, not field position) — no ExternalDNS, no propagation delays - `--port "22:22@loadbalancer"` on cluster create for SSH specs - A fresh admin PAT minted per-instance via `gitlab-rails runner` (shared CI tokens return 401 against fresh deployments; the token grep pattern must allow `.` for the GitLab 17+ routing suffix) - `GIT_CLONE_PATH` must be a literal path in child pipelines (gitlab-org/gitlab#220047) ## Goal / end state - All Kubernetes-version cluster tests run in per-job k3d clusters — no shared state, no standing infrastructure, no agent plumbing. - A `k3d:` trigger job holds a `parallel:matrix` with one row per Kubernetes version (`K3D_K8S_IMAGE` + `K3D_K8S_VERSION_SLUG`), each spawning a child pipeline with self-contained deploy + QA jobs. **Adding or removing a Kubernetes version = editing a matrix row.** - The primary k3d version runs automatically on MR pipelines (no manual `approve_cluster_tests` gate needed — jobs are ephemeral and contention-free); other versions are manual on MRs, automatic on nightly/stable pipelines. - Exactly **one real-cluster test remains: OpenShift at the latest supported version (currently 4.21)**, still behind the manual approval gate. No GKE, no vcluster, no EKS. - All other clusters are decommissioned (see below). ## Operator-specific adaptations (vs. the chart work) - Deployment goes through `scripts/test.sh` (kustomize overlays in `config/test/overlays/`, operator manifest install, GitLab CR apply) — not the chart's autodevops flow. `test.sh` needs a k3d mode: skip the agent context switch, skip the ExternalDNS annotation injection, and derive `DOMAIN` from nip.io. - TLS: the current jobs rely on a pre-provisioned wildcard secret (`TLS_SECRET=gitlab-ci-tls`). In k3d, the GitLab instance is served over plain HTTP on the nip.io domain, matching the chart's k3d jobs (`k3d.ingress.values.yaml`). cert-manager is still installed in-cluster (`task install_certmanager`, as the vcluster path does) — not for instance TLS, but because the operator's webhook certificate requires it (`manager.webhook.selfSignedCert.create` defaults to true) and a fresh k3d cluster has nothing pre-installed. - External services (Valkey, CloudNativePG, Garage) are already solved: `scripts/dev_dependencies.sh` sources the chart's CI lib scripts (`CHART_CI_LIB_REF`) and works identically inside k3d. - The `other_chart_versions` chart-version child pipelines also run cluster tests, so k3d bridges inside them mean **two levels of pipeline nesting** — supported by GitLab, but must be validated explicitly. - GitLab environments, `stop_review_*` jobs, `auto_stop_in`, and resource groups disappear for k3d jobs — the cluster dies with the job. ## Delivery sequence and status The first MR (single GKE version) must merge before anything else proceeds. Issues 3–5 can run in parallel once the matrix (issue 2) is in place; use MR dependencies like the chart work did. The MR column is filled in as MRs are opened. | # | Issue | MR | Status | Description | |---|-------|----|--------|-------------| | 1 | gitlab-org/cloud-native/gitlab-operator#2107 | gitlab-org/cloud-native/gitlab-operator!1533 | ✅ | PoC — migrate one GKE version (1.34) to k3d | | 2 | gitlab-org/cloud-native/gitlab-operator#2108 | gitlab-org/cloud-native/gitlab-operator!1562 | 🚧 | k3d version matrix — migrate remaining GKE versions (1.33, 1.35 / Gateway API) | | 3 | gitlab-org/cloud-native/gitlab-operator#2109 | | ⏳ | Migrate vcluster variants (1.35, flux/Envoy Gateway) and remove vcluster from CI | | 4 | gitlab-org/cloud-native/gitlab-operator#2110 | | ⏳ | Manual full-suite QA jobs on k3d | | 5 | gitlab-org/cloud-native/gitlab-operator#2111 | | ⏳ | Consolidate OpenShift to the latest supported version (4.21) only | | 6 | gitlab-org/cloud-native/gitlab-operator#2112 | | ⏳ | Decommission retired clusters | ## Decommissioning scope (issue 6 — checklist only, no automation yet) - GKE clusters: `operator133-ci-cluster`, `operator134-ci-cluster`, `operator135-ci-cluster` - OpenShift clusters: 4.18, 4.19, 4.20 (`ocp-ci-418/419/420`) - vcluster host cluster: `operatorvc-ci-cluster` - Agent registrations in `gitlab-org/distribution/infrastructure/kube-agents` - DNS records and wildcard certificates for the retired `k8s-ft.win` subdomains - `DOMAIN_*` / `VCLUSTER_*` CI/CD variables in `.gitlab-ci.yml` ## Acceptance criteria - [ ] All GKE and vcluster cluster-test jobs replaced by per-job k3d equivalents - [ ] Adding a new Kubernetes version test requires only a new `parallel:matrix` row - [ ] OpenShift 4.21 is the only remaining real-cluster test - [ ] Smoke QA runs automatically on MR pipelines for the primary k3d version - [ ] Retired clusters and their supporting infrastructure are decommissioned