Docker+machine: Fix VM leak when shutdown races a machine create or remove

Why

shutdown_drain (!6330 (merged)) leaks cloud VMs when shutdown races an in-flight create or remove. machineProvider.create() and machineProvider.remove() both spawn a goroutine that owns the machine for tens of seconds, and drain had no synchronisation with either:

  • Removing: drain skipped these and counted them as success on the assumption the goroutine would finish on its own. It often didn't: the process exited mid-docker-machine rm and the VM stayed running.
  • Creating: drain ran ForceRemove against an entry whose Driver state on disk wasn't fully written yet. For post-hoc-placement creates (bulkInsert with a flexibility policy) the ResolvedZone is empty pre-discovery and the rm subprocess constructs a delete with the operator-default zone; GCP rejects it, rm --force swallows the error and exits 0, drain logs success, VM stays running.

Reproduces in a multi-replica deployment by rolling mid-pipeline.

What

machineDetails gains a completion channel that create() / remove() set before spawning their goroutine and the goroutine closes on exit. Drain waits on the channel for Creating / Removing entries and drains Idle / Used / Acquired directly as before. On context expiry it logs the leak and counts the entry as failed in the drain summary instead of reporting success.

Edited by Igor

Merge request reports

Loading