Draft: feat: implicit spillover between workers
Draft — exploratory, for tracking and CI feedback.
Implicit, chainable spillover between workers. A secondary worker only requests jobs once every primary it lists is currently saturated (i.e. its most recent provider.Acquire() returned *NoFreeExecutorError, or it hit its concurrent-build limit). Motivating case: GCP zonal/project stockouts on gitlab.com, where we want a smaller idle pool in a fallback zone to absorb overflow without paying steady-state cost.
[[runners]]
name = "secondary"
[runners.spillover]
from = ["primary"]
hold = "30s" # treat primary as still saturated this long after its last NoFreeExecutorChainable: tertiary.from = ["secondary"], secondary.from = ["primary"]. Cycles, self-references, and unknown names are rejected at config-load.
Gate lives in gitlab-runner, not docker-machine: docker-machine has no notion of other pools, and gating above the executor keeps it executor-agnostic — both docker+machine and the autoscaler/fleeting executor already surface saturation as *NoFreeExecutorError.
Destination declares the relationship, not the source. Same convention as systemd After=, GitLab CI needs:, docker-compose depends_on. Fan-in works naturally, edits stay local, primary stays oblivious.
New metrics: gitlab_runner_saturated (gauge), gitlab_runner_saturation_events_total{reason} (counter, broken down by limit_hit vs no_capacity), gitlab_runner_spillover_events_total (counter, ticks only when a gated worker actually picks up a job), gitlab_runner_spillover_feed_skipped_total (counter, ticks when the gate holds a worker back). Pairs with the existing gitlab_runner_autoscaling_actions_total{action="creation-failed"} SLI. Info-level transition logs on every gate flip per worker.
Zero-value behaviour (no [runners.spillover] block) is identical to today.
Open questions before un-drafting
- Whether
NoFreeExecutorErrorshould carry aReasonfield so we can split it further (currently the in-treemarkSaturatedcall sites inferlimit_hitvsno_capacityfrom which path returned the error — works, but a per-error reason would be more robust). - Pre-warm tradeoff on the secondary is the Flex-MIG tradeoff — chef-repo knob, no code change.