Addressing GCP stockouts

Summary

  • Architecture: gitlab.com's Linux CI fleet runs on docker+machine. Each runner manager has one worker pinned to a single GCP project and zone via MachineOptions, and provisions job VMs through the GCE driver (compute.instances.insert).
  • Shard layout: A shard spans multiple projects for blast-radius and quota isolation; each project pinned to one zone. The small-amd64 shard (our largest) has 9 projects evenly split across us-east1-{b,c,d} at 3-3-3, so a zonal stockout removes a third of the shard's capacity and the remaining zones don't absorb the load.
  • Symptom: ZONE_RESOURCE_POOL_EXHAUSTED blocks job VM creation. Affected manager can't grow its pool, jobs queue at GitLab, no automated mitigation.
  • Deprecation context: docker+machine is deprecated upstream. Long-term direction: Kubernetes with gvisor/firecracker (Hosted Runners Platform v5). docker-autoscaler + fleeting is a backup, not the destination.

Background

Recent stockout incidents on the small-amd64 shard (production#21936 (closed), production#21939, production#21979) impacted gitlab.com CI job pickup. GCP's recommendation is Flex MIG (or instances.bulkInsert with instanceFlexibilityPolicy) — a precedence list of machine families that taps an internal obtainability mechanism unavailable to direct Instances.Insert retries.

InstanceFlexibilityPolicy.InstanceSelections[] also supports per-selection disks overrides (documented here), so a single bulkInsert can span machine families with different disk-type requirements (e.g. N2D on pd-balanced + N4 on hyperdisk-balanced). This invalidates the earlier assumption that cross-generation capacity rescue required a separate runtime fallback layer.

Options

Option 1: self-implemented spillover between workers — rejected

GCP confirmed that self-implemented retries don't tap the internal obtainability mechanism. Closed at gitlab-org/gitlab-runner!6726 (closed).

Option 2: docker-machine + RegionInstances.BulkInsert — landing

Modified the GCE docker-machine driver to provision via RegionInstances.BulkInsert with LocationPolicy + InstanceFlexibilityPolicy in the request body. Same internal-fulfillment path as Flex MIG without the external resource. Docker-machine stays the autoscaler; existing metrics and SLI (gitlab-org/gitlab-runner!6740 (merged)) keep working.

Option 3: migrate to docker-autoscaler + MIG (contingency)

Move executor + autoscaler + plugin + IAM + helper-image distribution + chef-repo at the same time as picking up Flex MIG. Loses target-label attribution (no fleeting metrics channel), and we've never run fleeting at gitlab.com scale. Contingency if we need to move off docker+machine before K8s is ready.

Risks

The latent bug we documented during the private-1 canary materialised at fleet-wide rollout as INC-10515 (see gitlab-org/ci-cd/docker-machine#148 (closed)). When the bulkInsert path fails after GCP has already placed the VM, the driver records no zone, and the runner-manager retries cleanup forever. The incident left 10,200 stuck machines and 257 orphan VMs. Driver-side self-heal lands in gitlab-org/ci-cd/docker-machine!175 (merged). The runner-side cap on the unbounded loop lands in gitlab-org/gitlab-runner!6791 (merged) as defence in depth.

Follow-ups

  • GPU shards on bulkInsert. Needs a per-selection accelerator override on InstanceFlexibilityPolicy.InstanceSelections[], shaped like docker-machine!173 but for Accelerators instead of Disks. Until then GPU jobs can't fall back across accelerator types and stay exposed to single-SKU stockouts.
  • ARM shards (saas-linux-*-arm64) on bulkInsert. The path is identical to amd64 once a flex selection family is settled (T2A / C4A). No driver changes needed, just chef-repo config in the corresponding parent roles.

Decisions

  • Run a controlled stockout comparison? Stockout impact is unmeasured.
  • Trigger conditions for 3 (currently contingency-only).
  • Owners per track: Runner Core (driver + runner-side MRs) vs Runners Platform (chef-repo, config-mgmt).
  • GCP quotas for bulkInsert: 24h Cloud Monitoring data shows current per-project peak of ~500-600/min on heavy_weight_read_requests_per_region (us-east1), which is the closest available proxy for what'll shift to OperationReadRequestsPerMinutePerProjectPerRegion after the switch (limit 4500/min, ~8x headroom). Inferred concurrent ops well under the 500 default bucket limit (~10x headroom). No preemptive raise; monitor on rollout and request if operation_read_requests_per_region climbs past ~50% of limit.
Edited by Igor