google: provision via RegionInstances.BulkInsert with flex policy

Igorrequested to merge
prototype/google-bulk-insert into main

Why

GCP zonal stockouts on n2-standard-2 are routinely impacting gitlab.com CI runners. Two axes of flexibility move the needle: zone fallback (try a different zone in the same region) and machine-type fallback (try n2d-standard-2 / c2-standard-4 when the preferred shape is unavailable). Both are doable on a single regional API call via RegionInstances.BulkInsert + InstanceFlexibilityPolicy, without any MIG resource to provision or own. A MIG-based variant was prototyped at !168 (closed) and closed in favour of this approach.

What

New explicit opt-in flag --google-bulk-insert selects the bulkInsert path. It pairs with --google-region (mutually exclusive with --google-zone). Two flags shape the request:

  • --google-flex-machine-type (repeatable; ranked by occurrence order, first = rank 0)
  • --google-location-zone (repeatable; zone[:PREFERENCE], PREFERENCE ∈ ALLOW/PREFERRED/DENY)

Opt-in is a separate flag rather than implied by --google-region so the provisioning channel is grep-able in config and on-disk Driver state. Operators using --google-region without --google-bulk-insert continue down the regular zonal Instances.Insert path.

LocationPolicy.TargetShape isn't exposed: at count=1 every shape collapses to "pick one zone". If a batched-create subcommand lands later, the flag comes back with the batched code.

Create is region-scoped (bulkInsert with count=1); the chosen zone is discovered post-create via AggregatedList and persisted on Driver.ResolvedZone. All other lifecycle ops stay zonal, keyed on ResolvedZone.

google.golang.org/api bumped from v0.169.0 to v0.279.0 to pick up BulkInsertInstanceResource.InstanceFlexibilityPolicy in the typed SDK.

Example

A [[runners.machine]] MachineOptions block exercising every new flag:

[runners.machine]
  MachineDriver = "google"
  MachineOptions = [
    "google-project=my-project",
    "google-machine-type=n2-standard-2", # ignored if google-flex-machine-type is set

    # bulkInsert path:
    "google-bulk-insert",
    "google-region=us-east1",

    # Machine-type fallback. Order = rank. GCE attempts n2-standard-2
    # first; if no zone has capacity for it, falls back to n2d, then c2.
    "google-flex-machine-type=n2-standard-2",
    "google-flex-machine-type=n2d-standard-2",
    "google-flex-machine-type=c2-standard-4",

    # Zone preference. PREFERRED zones are tried before ALLOW;
    # DENY zones are excluded entirely. Omit a zone to fall back to
    # GCE's default selection for that region.
    "google-location-zone=us-east1-b:PREFERRED",
    "google-location-zone=us-east1-c:ALLOW",
    "google-location-zone=us-east1-d:ALLOW",
  ]

The flex-* and location-* flags are optional. The minimal opt-in is --google-bulk-insert + --google-region; in that case GCE picks any zone in the region and uses --google-machine-type as-is (no fallback).

Other zone-bound flags (--google-machine-type, --google-disk-type, --google-accelerator) work as expected: GCE resolves bare resource names against the chosen zone at placement time.

Interaction between --google-machine-type and --google-flex-machine-type: if flex is set, the flex list fully owns machine-type selection and --google-machine-type is shadowed (warning logged when non-default). If flex is not set, --google-machine-type is used as-is with no stockout fallback (warning logged so operators notice they could be using flex).

Limitations

Two features the direct path supports don't work under bulkInsert, both because placement is post-hoc:

  • --google-address (static external IP): rejected at flag-validation.
  • Pre-existing disk reuse (the direct path's lookup for a disk named <instance-name>-disk): silently skipped, since the lookup needs a zone. Not exposed as a flag.

Related

Edited by Igor

Merge request reports