Loading
Extend docker+machine creation histogram buckets
What does this MR do?
Extends the bucket ranges of two docker+machine autoscaling histograms in executors/docker/machine/provider.go:
gitlab_runner_autoscaling_machine_creation_duration_seconds:ExponentialBuckets(30, 1.25, 10)→ExponentialBuckets(30, 1.25, 14), raising the top finite bucket from ~223s to ~546s.gitlab_runner_autoscaling_machine_failed_creation_duration_seconds:{1, 3, 5, 10, 30, 50, 60, 80, 90, 120}→{1, 5, 10, 30, 60, 120, 240, 480, 900}.
Why was this MR needed?
When cloud provider capacity is constrained, machine creation slows down and both histograms saturate into +Inf exactly when the tail matters:
- Successful creates that fall back across placement candidates can exceed the 223s ceiling, making p99 unreadable during capacity events. In one recent event on our fleet, 2.4% of successful creates overflowed the top bucket.
- Failed creates that exhaust placement fallback and then hit SSH/provisioning timeouts routinely exceed 120s; in the same event 8.4% of failure durations overflowed. That hides how long failed creates occupy
MaxGrowthRatecreation slots, which is the number you need when diagnosing replenishment throughput.
Adding buckets is backward compatible: existing histogram_quantile queries, dashboards, and recording rules keep working, and windowed rates handle the version boundary.
What's the best way to test this MR?
go test -race ./executors/docker/machine/ -count=1 passes. Scrape /metrics from a runner with a docker+machine executor and confirm the new le values appear on both histograms.
What are the relevant issue numbers?
- Incident: INC-13505 / gitlab-com/gl-infra/production#22786 (closed) — the capacity event where both histograms saturated (2.4% / 8.4% overflow)
- Proposal and analysis (confidential): gitlab-com/gl-infra/production-engineering#29633
Edited by Igor