Add runner-side Job Router metrics

Vishal Takrequested to merge
vtak/job_router_metrics into main

What does this MR do?

Adds Runner-side Prometheus metrics for the Job Router path. Until now the Runner only exposed the circuit-breaker metrics (gitlab_runner_job_router_circuit_breaker_*); this MR adds observability for the discovery cache, fallback behaviour, and per-Runner GetJob latency, bringing the Runner side closer to parity with the metrics already available on KAS.

Three metrics are added to router.Client (already registered as a prometheus.Collector), under namespace gitlab_runner, subsystem job_router:

Metric Type Description
gitlab_runner_job_router_discovery_cache_events_total{result="hit"|"miss"} Counter Discovery-cache lookups, partitioned by hit/miss - validates the cache is working.
gitlab_runner_job_router_fallbacks_total{reason=...} Counter Fallbacks from the router to direct GitLab polling, partitioned by reason: no_discovery, breaker_open, dial_failed, breaker_tripped, router_disabled. Confirms circuit-breaker fallback in production.
gitlab_runner_job_router_get_job_duration_seconds Histogram Runner-side GetJob request latency, observed on every call regardless of outcome - detects silent performance degradation.

Why was this MR needed?

Without Runner-side metrics, operators cannot validate discovery-cache functionality, detect silent GetJob performance degradation, or confirm circuit-breaker fallback operations in production. KAS already has equivalent metrics; the Runner was the blind spot.

What's the best way to test this MR?

  1. Enable the FF in Rails by running Feature.enable(:job_router).
  2. Build from this branch, register against an instance with the Job Router enabled, and set FF_USE_JOB_ROUTER=true and --listen-address (env LISTEN_ADDRESS) so /metrics is scrapeable.
  3. Run some CI jobs, then curl <listen-address>:<metrics-port>/metrics and confirm the three gitlab_runner_job_router_* series appear and increment.
  4. Disable the router (or trip the breaker) and confirm fallbacks_total increments with the expected reason label.

What are the relevant issue numbers?

Edited by Vishal Tak

Merge request reports