Draft: feat: implicit spillover between workers

Draft — exploratory, for tracking and CI feedback.

Implicit, chainable spillover between workers. A secondary worker only requests jobs once every primary it lists is currently saturated (i.e. its most recent provider.Acquire() returned *NoFreeExecutorError, or it hit its concurrent-build limit). Motivating case: GCP zonal/project stockouts on gitlab.com, where we want a smaller idle pool in a fallback zone to absorb overflow without paying steady-state cost.

[[runners]]
  name = "secondary"
  [runners.spillover]
    from = ["primary"]
    hold = "30s"   # treat primary as still saturated this long after its last NoFreeExecutor

Chainable: tertiary.from = ["secondary"], secondary.from = ["primary"]. Cycles, self-references, and unknown names are rejected at config-load.

Gate lives in gitlab-runner, not docker-machine: docker-machine has no notion of other pools, and gating above the executor keeps it executor-agnostic — both docker+machine and the autoscaler/fleeting executor already surface saturation as *NoFreeExecutorError.

Destination declares the relationship, not the source. Same convention as systemd After=, GitLab CI needs:, docker-compose depends_on. Fan-in works naturally, edits stay local, primary stays oblivious.

New metrics: gitlab_runner_saturated (gauge), gitlab_runner_saturation_events_total{reason} (counter, broken down by limit_hit vs no_capacity), gitlab_runner_spillover_events_total (counter, ticks only when a gated worker actually picks up a job), gitlab_runner_spillover_feed_skipped_total (counter, ticks when the gate holds a worker back). Pairs with the existing gitlab_runner_autoscaling_actions_total{action="creation-failed"} SLI. Info-level transition logs on every gate flip per worker.

Zero-value behaviour (no [runners.spillover] block) is identical to today.

Open questions before un-drafting

  • Whether NoFreeExecutorError should carry a Reason field so we can split it further (currently the in-tree markSaturated call sites infer limit_hit vs no_capacity from which path returned the error — works, but a per-error reason would be more robust).
  • Pre-warm tradeoff on the secondary is the Flex-MIG tradeoff — chef-repo knob, no code change.
Edited by Igor

Merge request reports

Loading