Docker+machine: Add discovered placement labels to autoscaling metrics
Adds provisioning-attribution labels (zone, region, project, architecture, machine type) to gitlab_runner_autoscaling_actions_total and gitlab_runner_autoscaling_machine_states. Values come from the actually-provisioned VM rather than operator-supplied configuration.
Why
Historically, runner managers were pinned per shard project to a single zone, and worker VMs landed in that zone alongside the manager. The manager's static configuration was both the placement contract and the source of truth for metric attribution: the two were synonymous because the manager was a 1:1 proxy for where workers ran.
bulkInsert with a flexibility policy decouples worker placement from manager pinning: a worker can land in any zone the policy allows, independent of where its manager runs. The same decoupling shows up in the runner-managers-on-Kubernetes direction: managers running in pods aren't pinned to any specific zone, so the manager's identity can't speak for worker placement either way. In both cases metric labels derived from the manager's static configuration would describe operator intent rather than worker reality. This MR is the metrics-side change: attribution moves to the worker itself, so per-(zone, machine type) failure rates stay meaningful.
Approach
Every create attempt contributes labels derived from what was actually provisioned. The runner reads the per-machine state file docker-machine writes during create, infers placement and machine type from it, and applies them on the metric counter. Failures included, so a stockout surfaces as creation-failure events labelled with the zone and machine type that got rejected. Also rerun when a manager rehydrates its in-memory state after restart.
Where the runner sources the label values depends on the provisioning mode recorded in the persisted driver state: post-hoc-placement modes use the actual placement; direct-create mode uses the operator's flag values. There is no operator-facing knob.
Other docker-machine drivers are not supported; the parsing is GCE-specific.
Related
- docker-machine!169: bulkInsert, writes the post-hoc-placement values this MR reads.
- runner-managers-on-Kubernetes design doc: same decoupling, different cause.
- production-engineering#29053: broader stockout-work tracking.