:dart: DRI: @treagitlab ### Participants * @treagitlab * @jay_mccure * @svistas # Local-to-remote provisioning: engineer-triggered remote Caproni environments + runbook ## Overview Let an engineer bring up a remote, vcluster-backed GitLab environment from their own machine — not only via CI — and have it be governed, reachable and reliably cleaned up. Caproni's `vcluster` driver already provisions, configures and deploys in one `caproni up --driver vcluster`. What is missing is the **shared substrate** engineers point it at, and the **documented path** to doing so without rediscovering everything by hand. ## Problem An engineer who wants a production-like GitLab running their change has local Caproni (colima/k3d), which is single-node and private, or CI, which they do not control interactively. There is no route to a remote environment they trigger themselves. It has been done once, by hand: GitLab + Orbit on a vcluster hosted on the `hangar-host` GKE cluster. That run exposed what is missing: - **No shared, governed substrate.** `hangar-host` exists but is not set up for multiple engineers — no per-user identity, no ownership guardrails, no base domain, no shared ingress. - **Reachability was hand-rolled.** The run exposed GitLab through a manually applied LoadBalancer Service, IP-locked to one person. That works for one engineer and does not generalise to a shared cluster or to sharing an environment with a colleague. - **Nothing cleans up.** An environment triggered from a laptop has no process that ends and tears it down. The host cluster from that first manual run sat autoscaled 3–7 nodes with no consumer, in a project outside this group's visibility, because nothing owned ending it — the failure mode in miniature. - **The path was undocumented.** Nothing on `main` described how the host is provisioned, governed, reached or reaped. ## What already exists Worth being explicit, because it makes this epic smaller than it first appears. - **The `vcluster` driver**, merged on Caproni `main`: provisions the vcluster in a host cluster, configures it (including TLS), and deploys CNG. Config surface is `cluster.vcluster.{host_context, namespace, chart_version, values, manage_namespace}`. - **A worked multi-tenant reference** in Caproni's [`examples/vcluster/`](https://gitlab.com/gitlab-org/caproni/-/blob/main/examples/vcluster/README.md): a Kyverno policy that stamps a `caproni.gitlab.io/owner` annotation on namespace creation, grants the creator update/delete rights and denies everyone else; tenant RBAC; a break-glass admin identity; and a shared host-level Traefik that routes to each tenant by TLS SNI over one IP, with a generated passthrough `IngressRouteTCP` for `<namespace>.<baseDomain>`. - **An install script** (`install-kyverno.sh`) that applies the policy, RBAC and base domain. - **`hangar-host`**, an existing GKE cluster that served the manual run. So the ownership guardrail and the shared-ingress design are already solved upstream. This epic is largely about standing them up for real, and writing down the route. ## Deliverables **1. A governed shared host cluster engineers can self-serve into** - Per-user identity at the host API server (GKE native IAM, or OIDC) so each engineer authenticates as themselves and the ownership policy has something to bind to - Kyverno, the ownership policy, tenant RBAC and the break-glass identity applied - A base domain and the shared host-level Traefik/SNI front, replacing per-environment hand-applied LoadBalancer Services - Confirmed delegated rights to create _and destroy_ in the hosting project, including leaked LoadBalancers, persistent volumes and disks — cleanup authority is useless without it **2. Minimal lifecycle control — a default TTL** - Every environment stamped with an expiry at creation, and a scheduled job that destroys expired ones. A 24-hour default is enough to start - This is deliberately _not_ the full governance control plane. It is the smallest thing that makes engineer-triggered environments safe to switch on, and it is days of work - Cost attribution labels on created resources, so spend can be traced per owner **3. The host provisioning runbook, published** - [`docs/provisioning-the-host.md`](https://gitlab.com/gitlab-org/quality/hangar-poc/-/blob/main/docs/provisioning-the-host.md) committed to `hangar-poc` alongside the caproni config it references, generalised beyond Orbit - **Two runbooks, agreed 2026-08-13.** The caproni team owns the consumer-facing onboarding runbook — how a team points caproni at a remote environment to build their service and GitLab. We own the substrate: how the host is provisioned, governed, reached and reaped. Most of the CLI and config friction from the manual run is caproni-product behaviour true for every team, so it goes to them rather than being duplicated here - Covers: access prerequisites, host cluster creation, the Kyverno guardrail, the ingress front, operating the reaper, and teardown in an order that actually releases cloud resources — with the consumer half linked, not copied - The known gotchas written down: caproni pinned to `v4.2.0`, which carries the `vcluster` driver with no source build needed; vcluster values paths resolved against the working directory rather than `caproni.yaml`; a cold GKE LoadBalancer takes \~75–80s before it routes; the base `extends` pulls `edit_mode_prepare` hooks that fail in a deploy-only configuration; deploy-only means no `repositories`/`reloaders`; `caproni --debug` rather than `LOG_LEVEL` ## Work breakdown and ordering Five children plus the project move, on two axes and documentation. **Where it lives** | | | | |--|--|--| | gitlab-org/quality&454 | Host cluster with per-user identity | **Closed 2026-08-13**, two identities verified | | [#4616](https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/work_items/4616) | Move the host off a personal sandbox into a team-owned GCP project | **Closed 2026-08-24**, host runs in `caproni-remote-envs-4fb1f3fd` | **Capability: isolation, lifecycle, routing** | | | | |--|--|--| | gitlab-org/quality&455 | Kyverno ownership guardrail and tenant RBAC | **Closed 2026-08-21**, peer isolation proven in both directions | | gitlab-org/quality&458 | Default 24h TTL, reaper, cost attribution | **Closed 2026-09-02** | | gitlab-org/quality&456 | Shared host-level Traefik SNI front and base domain | **Closed 2026-09-16**, ran live with two tenants.| **Documentation** | | | | |--|--|--| | gitlab-org/quality&459 | Publish the host provisioning runbook to `hangar-poc` | **Closed 2026-09-16**| **What is left:** gitlab-org/quality&456's live run, then gitlab-org/quality&459. ## Dependencies No external gate remains on this epic. - **Not dependent on** the CNG-orchestrator/CI work, the modular feature reference epic, or the full fleet governance control plane. ## Out of scope - **Edit mode.** No change assumed — a Caproni concern, not this epic's. - **The full fleet governance control plane** — group-owned namespaces, one namespace per team, the orphan sweep, registry, quotas, idle detection, cost reporting. [&463](https://gitlab.com/groups/gitlab-org/quality/-/work_items/463). This epic delivers only a default TTL, and it is scoped for one engineer per namespace; gitlab-org/quality&463 is what makes the host work for teams. - **Where the host permanently lives**, plus billing and the reaper's cloud IAM. [&457](https://gitlab.com/groups/gitlab-org/quality/-/work_items/457). The current project carries the POC and early onboarding. - **MR-triggered environments.** [&464](https://gitlab.com/groups/gitlab-org/quality/-/work_items/464). This epic is the engineer-triggered path; that one adds the MR trigger and sharing, and reuses this substrate. - **Replacing `staging-ref`.** Different problem, different parity bar. - Owning any long-lived named environment — engineers create their own. ## Exit criteria - [x] An engineer with no prior setup follows the published runbook and stands up their own environment. Done 2026-09-16: the first tenant worked through [`standing-up-a-vcluster.md`](https://gitlab.com/gitlab-org/quality/hangar-poc/-/blob/main/docs/standing-up-a-vcluster.md) cold and filed eight findings, all fixed in [hangar-poc!44](https://gitlab.com/gitlab-org/quality/hangar-poc/-/merge_requests/44). Putting GitLab in that environment is [`deploying-gitlab.md`](https://gitlab.com/gitlab-org/quality/hangar-poc/-/blob/main/docs/deploying-gitlab.md), run twice on this host the same day - [x] Two engineers can hold environments on the same host cluster simultaneously without being able to affect each other's - [x] Each environment is reachable at a stable per-environment hostname over TLS, with no hand-applied LoadBalancer. Two environments served GitLab behind the shared front at once on 2026-09-16, each with its own wildcard; four LoadBalancers on the host, all chart- or caproni-owned - [x] An environment left alone is destroyed automatically within its TTL, and the destruction leaves no orphaned LoadBalancers, volumes or disks - [x] Host provisioning runbook committed to `hangar-poc`, with the caproni team's consumer-facing runbook linked rather than duplicated <!-- STATUS NOTE START --> ## Status 2026-09-10 :clock1: **total hours spent this week by all contributors**: 28 :status: TL;DR: On track. The security question that was holding up the rest of the work was satisfied, and the shared entry point every environment will sit behind is live. :tada: **achievements**: - The access question is settled and AppSec has given verbal approval https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/work_items/4627#note_3805960582. Environments are reachable only from the company VPN and nothing is exposed to the internet. This was last week's blocker. The threat model merge request https://gitlab.com/gitlab-com/gl-security/product-security/appsec/threat-models/-/merge_requests/88 stays open but is no longer a condition for closing anything. - The shared entry point is running: one address serving every environment, restricted to the VPN gateways - [how it is installed and locked down](https://gitlab.com/gitlab-org/quality/hangar-poc/-/merge_requests/40). Verified from both sides, including from outside the VPN, where connections time out rather than being refused. :issue-blocked: **blockers**: - :arrow_forward: **next**: - The runbook is being tested by an engineer from outside the project - issue to track testing progress gitlab-org/quality/hangar-poc#1. That is the real test of whether the runbook is complete. - Confirm an environment answers on a stable name over HTTPS. That is the last substantive piece of &456. - Finish the runbook trial above and fold any feedback in &459. - Once the environments can be demonstrated end to end, take the hosting and billing ask to management &457. That is the last piece of &450 that needs a decision from outside the team. _Copied from https://gitlab.com/groups/gitlab-org/quality/-/epics/450#note_3812242810_ <!-- STATUS NOTE END -->