Draft: docker+machine: MachineOptionsFallback to retry instance creation across disk-type cohorts

Why

GCE's bulkInsert + InstanceFlexibilityPolicy (used in docker-machine!169) can't mix pre-hyperdisk and post-hyperdisk machine families in a single request. The disk type is fixed at the request level and the two generations require incompatible disk types: N4 strictly needs hyperdisk-balanced, N2D strictly needs pd-balanced. So a single bulkInsert can't fall back from an N4-stocked-out fleet to N2D for capacity rescue. Verified empirically against GCP; details in production-engineering#29053.

This matters operationally: gitlab.com just migrated to N4 fleet-wide. Our CUDs are on N2/N2D, so the families we have commercial commitment for are precisely the ones we can't reach via flex from an N4 fleet.

What

A runner-level fallback knob. When the primary MachineOptions create fails, the runner cleans up local state and retries with FallbackMachineOptions. Any failure triggers fallback; we don't try to classify whether the error is capacity-related, because pattern-matching subprocess stderr is brittle and the cost of wasting a retry on a non-recoverable failure (auth, validation) is small. Empty FallbackMachineOptions disables the feature; existing single-rung behaviour is preserved.

Operator config (TOML):

[[runners]]
  [runners.machine]
    MachineDriver = "google"
    MachineName = "runner-%s"
    MachineOptions = [
      "google-project=...",
      "google-machine-type=n4-standard-2",
      "google-bulk-insert",
      "google-region=us-east1",
      "google-flex-machine-type=n4-standard-2",
      "google-flex-machine-type=c4-standard-4",
      "google-disk-type=hyperdisk-balanced",
    ]
    FallbackMachineOptions = [
      "google-project=...",
      "google-machine-type=n2d-standard-2",
      "google-bulk-insert",
      "google-region=us-east1",
      "google-flex-machine-type=n2d-standard-2",
      "google-flex-machine-type=n2-standard-2",
      "google-disk-type=pd-balanced",
    ]

Full replacement (verbose, but explicit). The fallback list is a complete options list, not a delta: operators see exactly what each rung will use.

Observability

Adds fallback-attempted / fallback-succeeded / fallback-failed values to gitlab_runner_autoscaling_actions_total, plus a log line on fallback success.

Limitations

• Only one fallback level. Multi-rung extension would mean an array-of-lists in TOML; deferred until we see a real need beyond two.
• Doesn't help when both rungs are simultaneously exhausted. That's a region-wide GCP capacity event with no runner-side fix.
• Retries the ENTIRE docker-machine create pipeline (VM creation + SSH key upload + Docker daemon setup + TLS cert distribution), not just the VM-creation step. SSH-stage failures (network blip during cert upload, Docker daemon slow to start) trigger a full pipeline re-run on a different rung, which is wasteful (the VM from primary already exists). To retry only the VM-creation step would require pushing fallback down into the docker-machine driver itself, splitting Create from Provision (--no-provision on create + separate provision call). Not in scope for this MR.
• Latency: a primary failure adds whatever time it took to fail, on top of the fallback create. How much depends on where the primary failed.
• Increased GCP Compute API operations. Every primary failure triggers a second bulkInsert + cleanup delete. At fleet scale, a sustained primary-failure rate doubles our API request volume against the same project's quota window.

Related

• docker-machine!171: driver-level variant of this MR. Pushes the fallback decision into the Google driver itself; scope is the VM-creation step only (no full pipeline re-run on SSH-stage failures).
• docker-machine!169: bulkInsert + flex policy, which this MR backstops for cross-generation cases.
• production-engineering#29053: broader stockout-work tracking.
• !6740 (merged): per-VM target_* labels (used here for fallback-rung attribution).