## Summary - **Architecture**: gitlab.com's Linux CI fleet runs on `docker+machine`. Each runner manager has one worker pinned to a single GCP project and zone via `MachineOptions`, and provisions job VMs through the GCE driver (`compute.instances.insert`). - **Shard layout**: A shard spans multiple projects for blast-radius and quota isolation; each project pinned to one zone. The small-amd64 shard (our largest) has 9 projects evenly split across us-east1-{b,c,d} at 3-3-3, so a zonal stockout removes a third of the shard's capacity and the remaining zones don't absorb the load. - **Symptom**: `ZONE_RESOURCE_POOL_EXHAUSTED` blocks job VM creation. Affected manager can't grow its pool, jobs queue at GitLab, no automated mitigation. - **Deprecation context**: `docker+machine` is deprecated upstream. Long-term direction: Kubernetes with gvisor/firecracker ([Hosted Runners Platform v5](https://gitlab.com/gitlab-com/gl-infra/ci-runners/-/work_items/1)). `docker-autoscaler` + fleeting is a backup, not the destination. ## Background Recent stockout incidents on the small-amd64 shard (production#21936, production#21939, production#21979) impacted gitlab.com CI job pickup. [GCP's recommendation](https://gitlab.slack.com/archives/C0B12JPQB0R/p1778696781085699?thread_ts=1777915692.263049&cid=C0B12JPQB0R) is Flex MIG (or `instances.bulkInsert` with `instanceFlexibilityPolicy`) — a precedence list of machine families that taps an internal obtainability mechanism unavailable to direct `Instances.Insert` retries. `InstanceFlexibilityPolicy.InstanceSelections[]` also supports per-selection `disks` overrides (documented [here](https://docs.cloud.google.com/compute/docs/instance-groups/about-instance-flexibility#overrides-for-instance-properties)), so a single bulkInsert can span machine families with different disk-type requirements (e.g. N2D on `pd-balanced` + N4 on `hyperdisk-balanced`). This invalidates the earlier assumption that cross-generation capacity rescue required a separate runtime fallback layer. ## Options ### ~~Option 1~~: self-implemented spillover between workers — **rejected** GCP confirmed that self-implemented retries don't tap the internal obtainability mechanism. Closed at gitlab-org/gitlab-runner!6726. ### Option 2: docker-machine + `RegionInstances.BulkInsert` — **landing** Modified the GCE docker-machine driver to provision via `RegionInstances.BulkInsert` with `LocationPolicy` + `InstanceFlexibilityPolicy` in the request body. Same internal-fulfillment path as Flex MIG without the external resource. Docker-machine stays the autoscaler; existing metrics and SLI (gitlab-org/gitlab-runner!6740) keep working. - https://gitlab.com/gitlab-org/ci-cd/docker-machine/-/merge_requests/169: bulkInsert + flex policy. - https://gitlab.com/gitlab-org/ci-cd/docker-machine/-/merge_requests/173: per-selection disk override on flex policy, lets one request span pd-balanced families and hyperdisk-balanced families. - https://gitlab.com/gitlab-org/ci-cd/docker-machine/-/merge_requests/175: orphan-VM cleanup after failed bulkInsert ([INC-10515](https://app.incident.io/gitlab/incidents/10515)). - https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/6740: per-VM `target_*` metric labels sourced from docker-machine's resolved state. - https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/6791: cap `finalizeRemoval` retries. ### Option 3: migrate to `docker-autoscaler` + MIG (contingency) Move executor + autoscaler + plugin + IAM + helper-image distribution + chef-repo at the same time as picking up Flex MIG. Loses target-label attribution (no fleeting metrics channel), and we've never run fleeting at gitlab.com scale. Contingency if we need to move off `docker+machine` before K8s is ready. ## Risks The latent bug we documented during the private-1 canary materialised at fleet-wide rollout as [INC-10515](https://app.incident.io/gitlab/incidents/10515) (see https://gitlab.com/gitlab-org/ci-cd/docker-machine/-/work_items/148). When the bulkInsert path fails after GCP has already placed the VM, the driver records no zone, and the runner-manager retries cleanup forever. The incident left 10,200 stuck machines and 257 orphan VMs. Driver-side self-heal lands in https://gitlab.com/gitlab-org/ci-cd/docker-machine/-/merge_requests/175. The runner-side cap on the unbounded loop lands in https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/6791 as defence in depth. ## Follow-ups - GPU shards on bulkInsert. Needs a per-selection accelerator override on `InstanceFlexibilityPolicy.InstanceSelections[]`, shaped like [docker-machine!173](https://gitlab.com/gitlab-org/ci-cd/docker-machine/-/merge_requests/173) but for `Accelerators` instead of `Disks`. Until then GPU jobs can't fall back across accelerator types and stay exposed to single-SKU stockouts. - ARM shards (`saas-linux-*-arm64`) on bulkInsert. The path is identical to amd64 once a flex selection family is settled (T2A / C4A). No driver changes needed, just chef-repo config in the corresponding parent roles. ## Decisions - Run a controlled stockout comparison? Stockout impact is unmeasured. - Trigger conditions for 3 (currently contingency-only). - Owners per track: Runner Core (driver + runner-side MRs) vs Runners Platform (chef-repo, config-mgmt). - GCP quotas for bulkInsert: 24h Cloud Monitoring data shows current per-project peak of ~500-600/min on `heavy_weight_read_requests_per_region` (us-east1), which is the closest available proxy for what'll shift to `OperationReadRequestsPerMinutePerProjectPerRegion` after the switch (limit 4500/min, ~8x headroom). Inferred concurrent ops well under the 500 default bucket limit (~10x headroom). No preemptive raise; monitor on rollout and request if `operation_read_requests_per_region` climbs past ~50% of limit.