Run docker-machine in its own process group

Igorrequested to merge
docker-machine-own-process-group into main

What does this MR do?

Starts docker-machine in its own process group, kills that group when the command's context is cancelled, and makes Exist stop reporting "gone" for an inspect that did not run to completion.

Why was this MR needed?

On the Kubernetes runner managers the pod's PID 1 is dumb-init in its default setsid mode, and the image's STOPSIGNAL is SIGQUIT. When a pod stops, dumb-init forwards SIGQUIT to the whole process group, so every in-flight docker-machine gets it as well as the runner. docker-machine has no signal handler; the Go runtime prints a goroutine dump and exits 2. On gitlab.com that is ~13k SIGQUIT: quit dumps in 7 days, all in rollout hours (8496 create, 3309 stop, 971 remove, 11 exists, 11 force-remove).

Most interrupted creates are cleaned up by the existing failure path (Machine creation failed → remove). Two cases leak the VM:

  • a create killed between the bulkInsert request and its response: the follow-up rm runs AggregatedList before GCP lists the instance, treats it as not found, and drops the local store entry while the VM keeps running;
  • an interrupted inspect behind Exist: any error returned false, the provider logged Skipping machine removal, because it doesn't exist, and dropped the entry. All 11 cases in 7 days were VMs that had just finished a build; the ones I checked were still running days later or had been removed by the external stale-VM cleaner.

Setpgid keeps init-forwarded group signals away from docker-machine regardless of the entrypoint. The runner's shutdown drain already waits for in-flight operations before removing machines. Cancelling the context now kills the group, so the driver plugin child that docker-machine spawns dies with its parent instead of being reparented.

Exist now returns false only when inspect completed and exited 1 (invalid store entry). A timeout, a signal, or exit 2 leaves the answer unknown, and treating unknown as "gone" is what leaks the VM. This also covers an inspect that hits machineExistCommandTimeout with no signal involved.

Reproduced in the runner sandbox with the chart's dumb-init default (IdleCount = 40, MaxGrowthRate = 10, kubectl rollout restart while creates are in flight). main: 7 pod stops, 29 interrupted creates, dumps and exit status 2 failures on every stop that had creates in flight. This branch: 3 pod stops with 2, 0, and 3 creates in flight, no dumps, no failed creates, Machine pool drain completed failed=0 each time, no VMs left after scaling to 0.

What's the best way to test this MR?

TestDockerMachineRunsInOwnProcessGroup, TestDockerMachineCancelKillsProcessGroup, and TestExist in helpers/docker. TestExist/inspect_crashes_(SIGQUIT_dump_exits_2), inspect_killed_by_a_signal, and inspect_times_out fail on main.

What are the relevant issue numbers?

Edited by Igor

Merge request reports