## The problem docker+machine fleets create raw GCE instances. Nothing on the GCP side ties an instance's lifecycle to the manager that created it: when a manager dies uncleanly or loses its machine store, its VMs keep running with nothing left to remove them, consuming quota and money. Instance age can't distinguish a leaked VM from a tracked one, because machine reuse (`MaxBuilds × (timeout + IdleTime)`) makes multi-day lifetimes legitimate. This design gives every ephemeral VM a liveness signal, an enforcement mechanism for when the signal goes stale, and cause-based alerts for the manager failures that produce orphans in the first place. An instance is live when a live manager tracks it. That is the property everything below measures and enforces. ## Design | Failure | Handled by | Alert | |---|---|---| | Store loss on container restart | Store persistence (apps!3292, shipped) | none | | Orphans from unclean pod loss (eviction, node death) or manager decommissioning | Heartbeat + reaper, below | reaper stopped sweeping, or budget blocked deletions | | Manager OOM / crash loop | Static limits, can't self-heal | deployment restarts ≥N in 30m | | Shard can't scale (quota, stockout, image breakage) | Flex-selection fallbacks, until they run out | `machine_creation` SLI | | Quota saturation | Existing `gcp_quota_limit` alerts | unchanged | The principle: prefer making the system absorb a fault over alerting on it. Alert on causes where there is one clear human action, or when the cleanup mechanism itself stops working. No new symptom alerts: the existing quota saturation alerts remain the final catch-all and should only fire for real capacity growth. ### The liveness signal (heartbeat) Each manager stamps a `runner_manager_heartbeat` label (unix seconds, in the existing `runner_manager_*` label family) on every instance it tracks. A stale label means no live manager tracks the instance, whatever the cause: wiped store, dead process, dead pod, decommissioned manager. - The label write is the docker-machine `update-labels` command, implemented in the google driver. gitlab-runner refreshes each tracked machine's label on `HeartbeatInterval`, spread over the interval with bounded concurrency, and keeps beating while all worker slots are busy. The two binaries release together, and one implementation covers the chef and k8s fleets. MRs: https://gitlab.com/gitlab-org/ci-cd/docker-machine/-/merge_requests/190, https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/7264 - The runner exports a timestamp metric for its last heartbeat pass, so a broken writer and mass orphaning look different to the reaper. - The writer has operational bounds: a configurable interval, label writes spread over the interval rather than bursting once per pass, and bounded write concurrency. Label writes are mutations sharing the per-project GCP operation quota with instance create/delete, so the heartbeat must never starve autoscaling. ### Enforcement (reaper) https://gitlab.com/gitlab-com/gl-infra/reaper, replacing `ci-project-cleaner`. A Go service: one Deployment sweeping every 10 minutes with a worker pool over the projects, Prometheus metrics in-process, the GCP SDK instead of gcloud. The reap decision (instance list + policy in, deletions out) is a unit-tested function, and every rule can be set to off, report, or delete independently. The budget refusal is safety logic and needs tests: the manual sweep scripts that motivated this needed three fixes for bash failure modes, including one that would have classified the entire live fleet as orphaned. The reaper deletes a `ci_ephemeral` instance when all of these hold: - heartbeat label present and stale for longer than the longest job timeout plus slack (instances without the label are out of scope and fall through to the age rule). The quarantine is measured on the label value, not instance age: a dead manager doesn't stop the workload it left behind, and the job's external effects (deploys, uploads) can keep running until the timeout. - per-project budget respected: if most of a project's instances go stale at once, the likely fault is the writer (IAM, API quota), so report instead of deleting, and alert. Only persisted heartbeats count: a failed write never advances the label. - the writer-health signal is clean: the reaper holds off while managers report failing heartbeat passes. Independent of the heartbeat, the reaper also deletes `TERMINATED` instances (they hold disks and count against quotas but can't be running jobs) and unlabeled instances past a conservative `P3D`, matching the old cleaner. Precision comes from the heartbeat; age stays the coarse fallback. **Rollout phases.** 1: heartbeat ships as telemetry only. 2: `TERMINATED` cleanup automated. 3: reaper report-only until quarantine and correlated-staleness behavior are validated against real data. 4: enforcement. **Observability.** Per-project census metrics, plus alerts if the reaper or the heartbeat writer stops running. ### Cause alerts (runbooks) - Crash-loop alert for runner-manager deployments in the `runner-managers-*` clusters: restarts ≥N in 30 minutes, any reason. Manager death is the event that produces orphans and fails in-flight jobs; today it is invisible until a downstream quota alert. - Promote the `machine_creation` SLI out of experimental. Every scaling failure surfaces there regardless of cause. It was left non-alerting pending a baseline, which we now have. Not adding: memory-headroom warnings (dashboard material, and manager memory should be sized from the shard's `concurrent` setting rather than usage lookbacks) and tracked-vs-actual divergence in Prometheus (the shared runner projects span four Mimir tenants, the reaper census measures the same thing from the GCP side). ### Other executors The heartbeat is docker+machine-specific by design, because only docker+machine creates unowned raw instances. Kubernetes-executor pods are single-use, so job timeout bounds their legitimate age, and the executor already sets `activeDeadlineSeconds` (on by default): the kubelet kills timed-out pods with no manager involved, and auxiliary resources are garbage-collected via ownerReferences. Fleeting-based executors reattach to their instance group and adopt existing instances on restart, so per-instance orphaning doesn't apply; the residual check is groups that still hold instances after their manager is decommissioned. ### Rejected alternatives - Ownership labels reconciled against pod existence: managers can orphan VMs while their pod is alive (wiped store), so pod existence proves nothing. - VM-side heartbeats: an orphaned VM is healthy and heartbeats forever. The writer must be the manager, because tracking is the property being measured. - Store-reading sidecar: no runner release needed, but it keeps heartbeating for a wedged runner process and chef needs a second implementation. Fallback if release timing slips. ## Background This started as the corrective action for [INC-13578](https://app.incident.io/gitlab/incidents/13578) (related: [INC-13572](https://app.incident.io/gitlab/incidents/13572)): OOM crash-looping managers lost their machine stores and abandoned ~8,000 VMs, exhausting two GCP quotas. The first alert came three hours in, from quota saturation, two causal steps downstream. Cleanup was a manual census and reap. The immediate fixes (memory sizing in apps!3291, store persistence in apps!3292) closed the specific orphan source; this design closes the class. ## Verification - Replay the crash-loop and machine-creation alerts against the incident window in Mimir. - Rerun the store-persistence sandbox test against the reaper: kill the manager, confirm the leaked VMs are reaped within one cycle, and confirm the budget blocks deletion when the heartbeat writer is stopped.