Loading
Export autoscaling limits as gauges from the machine provider
What does this MR do?
Adds two gauges to the docker+machine provider, labeled by runner, runner_name, and system_id:
gitlab_runner_autoscaling_max_growth_rate: the configuredMaxGrowthRate.gitlab_runner_autoscaling_idle_target: the number of available machines the autoscaler currently aims for, after applyingIdleScaleFactorandIdleCountMin.
Both are updated on every Acquire pass, next to the existing per-state machine counts.
Why was this MR needed?
The provider exports machine activity but not the limits that activity runs against. To tell whether a manager is at its creation ceiling or its idle pool is draining, operators look the limits up in fleet configuration and hardcode them into dashboards, where they go stale on every tuning change. During INC-13505, runner managers sat at their creation ceiling for about four hours before the effect showed up in job-level SLIs, and confirming the ceiling meant reading fleet configs by hand.
The idle target is exported as the computed value rather than the raw settings because the value the creation loop aims for depends on the scale factor, the minimum, and current usage. Exporting the raw settings would push that computation into every dashboard.
What does this enable?
Ceiling saturation as a division:
sum by (instance) (gitlab_runner_autoscaling_machine_states{state="creating"})
/ on(instance) group_left sum by (instance) (gitlab_runner_autoscaling_max_growth_rate)and the same with state="idle" over gitlab_runner_autoscaling_idle_target for the idle pool. When a limit is tuned fleet-wide, each manager's gauge steps as it picks up the change, so incident timelines can use the metric instead of merge timestamps.
What's the best way to test this MR?
On a running docker+machine manager, curl localhost:9252/metrics | grep autoscaling shows the two new gauges next to the machine states.
What are the relevant issue numbers?
Edited by Igor