google: Per-selection disk override on flex policy

Igorrequested to merge
experiment/flex-per-selection-disk-override into main

Summary

Replaces --google-flex-machine-type (introduced in !169 (merged)) with --google-flex-selection, a comma-separated key=value list. Required key machine-type; optional disk-type, disk-iops, disk-throughput attach a per-selection disk override to InstanceFlexibilityPolicy.InstanceSelections. One bulkInsert request can now flex across machine families that require different disk types and IOPS/throughput settings.

--google-bulk-insert now requires at least one --google-flex-selection. The implicit fallback to --google-machine-type is gone; bulkInsert is explicitly a flex-mode opt-in.

What

[runners.machine]
MachineOptions = [
  "google-project=...",
  "google-bulk-insert",
  "google-region=us-east1",
  "google-disk-type=pd-balanced",
  "google-flex-selection=machine-type=t2d-standard-2",
  "google-flex-selection=machine-type=n2d-standard-2",
  "google-flex-selection=machine-type=n2-standard-2",
  "google-flex-selection=machine-type=n4d-standard-2,disk-type=hyperdisk-balanced,disk-iops=3000,disk-throughput=140",
  "google-flex-selection=machine-type=n4-standard-2,disk-type=hyperdisk-balanced,disk-iops=3000,disk-throughput=140",
]

Entries with only machine-type inherit the template disk (here, pd-balanced). Entries with disk-type attach an override that GCP merges with the template by deviceName, replacing the boot disk on that selection.

Why

Hosted runners have been hitting recurring capacity stockouts in us-east1 on the families we have GCP CUDs for (T2D/N2D/N2, on pd-balanced). When that happens we want to fall back to N4D/N4 (newer, more expensive, on hyperdisk-balanced): paying more for capacity beats having no capacity.

A previous design assumed bulkInsert couldn't mix pd-balanced and hyperdisk-balanced families in one request because disk types are pinned per request (see !171 (closed), closed in favour of this). That turns out to be wrong: GCP's InstanceFlexibilityPolicy.InstanceSelections[].disks lets each selection carry its own disk override, documented at https://docs.cloud.google.com/compute/docs/instance-groups/about-instance-flexibility.

Limitations

  • Selection-disk override inherits the template's image, size, and labels. Per-selection overrides for those would need additional keys.
  • Validation is light: GCP rejects machine-type/disk-type mismatches at submit, so we don't pre-check them on our side.
  • Per-selection disk/minCpuPlatform overrides are in Preview at GCP: pre-GA terms apply, behaviour can shift, support is limited.

Related

Edited by Igor

Merge request reports