Run a synthetic job at startup before reporting Ready

Why

The chart's default probes are pgrep gitlab-runner (process alive) and gitlab-runner verify (token valid). Neither catches the case where the manager can authenticate but can't provision or dispatch work: broken service-account permissions, a misconfigured driver, missing network egress, a broken bulkInsert region. A pod in that state reports Ready, accepts jobs, and fails every one of them.

The runner-managers-on-Kubernetes design doc names this gap under "deeper health checks".

The VM-based deployment via ci-runners/deployer already does a version of this: every linux runner-manager start provisions a test VM, SSHes in, and checks Docker. On Kubernetes that harness is gone. This is the in-process equivalent, and it also catches runner-side dispatch failures the VM check can't see.

What

Once per process start, the runner dispatches a synthetic job per enabled [[runners]] section through the normal build pipeline. If every job succeeds, a new /health/ready endpoint returns 200. If any fails, the runner drains in-flight VM creates and exits non-zero so kubelet restarts the pod.

It runs at boot only. Config reloads don't re-run it, and rollouts spawn fresh pods anyway.

Because the check is a regular job, not an executor-specific hook, it works for any executor: docker+machine, docker-autoscaler, kubernetes, shell, ssh.

On failure it waits for pending VM creates to finish so it can clean them up, then exits non-zero. This relies on !6754 (merged). Without it, VMs already submitted to bulkInsert leak when the runner gives up.

Config

Opt in per runner with a [runners.experimental.boot_verify] section. Only enabled is required. The timeouts default to the values shown and are documented in advanced-configuration.md.

The feature ships as an Experiment: the config format, the /health/ready endpoint, and the feature itself may change or be removed in any release. Nesting the config under [runners.experimental] makes that structural (mirroring the global [experimental] section that hosts usage_logger), so nobody adopts it without noticing. When it stabilizes — likely after the job router lands and the push-based design is settled — it graduates to [runners.boot_verify].

[runners.experimental.boot_verify]
  enabled = true
  timeout = "5m"               # per-runner deadline
  acquire_min_backoff = "1s"   # executor acquire retry backoff
  acquire_max_backoff = "10s"

Billing

Synthetic builds carry a Build.Synthetic flag, and the usage-log write skips tagged builds. The canary also never reaches that path today, because it calls build.Run directly instead of going through real-job processing. The flag is defense in depth: synthetic jobs stay out of billing even if a future refactor routes them through the real-job path.

Companion chart MR

The existing readiness probe goes Ready almost immediately, so it can't gate a rollout on the check. gitlab-org/charts/gitlab-runner!551 (merged) adds an optional startupProbe pointing at /health/ready.

Related

• !6740 (merged): per-VM provisioning labels.
• !6754 (merged): in-flight drain semantics, required for the clean failure-exit path.
• gitlab-org/ci-cd/docker-machine!169 (merged): the bulkInsert prototype this path exercises end-to-end.