google: clean up orphan VMs left by failed bulkInsert
Bug
When a bulkInsert request fails after GCP has already placed the VM (for example with VM_MIN_COUNT_NOT_REACHED), the driver loses track of which zone the VM ended up in. Every subsequent cleanup attempt is rejected by GCP for not specifying a zone, and the runner-manager has no way to tell that the situation is unrecoverable, so it retries forever. During INC-10515 this left 10,200 stuck machines and 257 orphan VMs.
Fix
If we don't know the zone at delete time, ask GCP. The driver re-runs the same lookup it would have done after a successful create. If the lookup finds the VM, we clean it up properly. If it doesn't, we tell the runner-manager the VM is already gone so it stops retrying.
This only kicks in for bulkInsert. In the classic direct-create flow an unknown zone is a real bug and should not be papered over.
Testing
I ran this in a sandbox runner-manager fleet with three of the production failure modes injected by another (sandbox-only) branch. Without the fix the fleet quickly accumulated dozens of machines stuck retrying, with tracked state diverging from the actual VMs in GCP. With the fix nothing stuck, tracked state stayed in step with GCP, and each failed create cleaned up in a few seconds without retries.
Out of scope
A separate runner-manager change to cap remove retries per machine, as defence in depth for failure classes the driver cannot self-heal.
References
- !148 (merged) / INC-10515
- Original bulkInsert MRs: !169 (merged) and !173 (merged)
- Operational revert from the incident: chef-repo!7431