Draft: google: Fallback machine and disk type

Igorrequested to merge
experiment/google-driver-fallback into main

Summary

Adds opt-in fallback flags to the Google driver. Primary createInstance failure triggers one retry against the fallback config.

What

Three opt-in flags: --google-fallback-machine-type, --google-fallback-disk-type, --google-fallback-flex-machine-type (repeatable). Any primary createInstance failure triggers the fallback (no error classification). Empty flags = no fallback. Scope is VM creation; SSH key generation, firewall setup, and post-create provisioning are not retried.

Why

Hosted runners have been hitting recurring capacity stockouts in us-east1 on the families we have GCP CUDs for (T2D/N2D/N2, on pd-balanced). When that happens we want to fall back to N4D/N4 (newer, more expensive, on hyperdisk-balanced): paying more for capacity beats having no capacity. bulkInsert can't mix them in a single request because disk types are pinned per request, so the flex policy inside one bulkInsert can only cover families that share a disk type.

This MR adds a second config the driver tries once if the primary fails, so operators can configure a primary across one generation and a fallback across another.

Why driver-level

An earlier version gitlab-org/gitlab-runner!6762 (closed) put the same fallback at the runner level, wrapping the whole docker-machine create invocation. That meant a transient SSH or Docker-daemon failure on the primary VM would also trigger a full pipeline re-run on the fallback rung: pointless, because SSH issues aren't a function of machine type and the primary VM already exists. Pushing the fallback into the driver scopes the retry to the VM creation step only.

Cost: GCE-specific. If we ever need the same mechanism on another driver, we'll add it there.

Example runner config

[[runners]]
  [runners.machine]
    MachineDriver = "google"
    MachineName = "runner-%s"
    MachineOptions = [
      "google-project=...",
      "google-bulk-insert",
      "google-region=us-east1",
      "google-disk-type=pd-balanced",
      "google-flex-machine-type=t2d-standard-2",
      "google-flex-machine-type=n2d-standard-2",
      "google-flex-machine-type=n2-standard-2",
      "google-fallback-disk-type=hyperdisk-balanced",
      "google-fallback-flex-machine-type=n4d-standard-2",
      "google-fallback-flex-machine-type=n4-standard-2",
    ]

Limitations

  • Google driver only.
  • One fallback level.
  • Sustained primary failures double the rate of create API calls.

Related

Edited by Igor

Merge request reports