Add circuit breaker for runner fallback to Rails on Relay failures
What does this MR do?
Adds a circuit breaker to the runner's Job Router path so a Relay outage fails over to direct Rails polling instead of stalling job pickup.
Why was this MR needed?
After the runner discovers the Job Router, it caches the router endpoint for up to discoveryTTL (1 hour).
Before this change, if Relay became unavailable after discovery, every GetJob call returned (nil, true) -
the runner looked healthy but picked up no jobs - for the remainder of the cache window.
A Relay outage therefore translated into a CI scheduling outage of up to an hour with no automatic recovery.
This was the highest-risk operational gap in the Job Router rollout.
What's the best way to test this MR?
Setup: a runner built from this branch, registered against a GitLab instance with the Job Router enabled, and the FF_USE_JOB_ROUTER feature flag on (config.toml [runners.feature_flags] or FF_USE_JOB_ROUTER=true and in Rails - Feature.enable(:job_router)). Set --listen-address (env LISTEN_ADDRESS) so you can scrape /metrics.
- Baseline - run a few CI jobs and confirm they're served via Relay.
curl <listen-address>/metrics | grep job_router_circuit_breakershowsstate 0, trips_total 0. - Induce a Relay outage after discovery (so the runner is in the cache window): stop the Relay service.
- Observe the trip - after 3 consecutive failed polls the runner logs Job router circuit breaker tripped, falling back to direct job requests (warn). Jobs keep being picked up via direct Rails polling. Metrics:
state 1, trips_total 1. - Confirm no stall - queue a job during the outage; it should still run via Rails, not hang until the 1h TTL.
- Restore Relay - bring the service/endpoint back. Within ~30s the next poll issues a half-open trial; on success the runner logs Relay recovered, resuming job router (info), state returns to 0, and routing resumes through Relay.