Docker+machine: Fix VM leak when shutdown races a machine create or remove
Why
shutdown_drain (!6330 (merged)) leaks cloud VMs when shutdown races an in-flight create or remove. machineProvider.create() and machineProvider.remove() both spawn a goroutine that owns the machine for tens of seconds, and drain had no synchronisation with either:
Removing: drain skipped these and counted them as success on the assumption the goroutine would finish on its own. It often didn't: the process exited mid-docker-machine rmand the VM stayed running.Creating: drain ranForceRemoveagainst an entry whose Driver state on disk wasn't fully written yet. For post-hoc-placement creates (bulkInsert with a flexibility policy) theResolvedZoneis empty pre-discovery and the rm subprocess constructs a delete with the operator-default zone; GCP rejects it,rm --forceswallows the error and exits 0, drain logs success, VM stays running.
Reproduces in a multi-replica deployment by rolling mid-pipeline.
What
machineDetails gains a completion channel that create() / remove() set before spawning their goroutine and the goroutine closes on exit. Drain waits on the channel for Creating / Removing entries and drains Idle / Used / Acquired directly as before. On context expiry it logs the leak and counts the entry as failed in the drain summary instead of reporting success.
Related
- !6330 (merged): the
shutdown_drainfeature this extends. - docker-machine!169: bulkInsert prototype where the Creating-race was first surfaced.
- runner-managers-on-Kubernetes design doc: rolling-update VM leakage matters specifically for the Kubernetes deployment shape this doc targets.
Edited by Igor