Docker+machine: cap finalizeRemoval retries

Igorrequested to merge
experiment/edge-case-fixes into main

Bug

When removeMachine returns a permanent error (one that retrying will not fix), finalizeRemoval retries it forever. One goroutine gets pinned per stuck machine and its IdleCount slot is never freed. This was the cost-driver behind INC-10515 on GitLab.com: 10,200 stuck machines across one shard during a 6-hour window, with the runner-managers spinning indefinitely against unresolvable state.

The most common trigger for a permanent error today is a docker-machine driver bug. docker-machine!175 fixes the trigger we have so far, but the runner is the layer that gets to decide how long to keep trying, and forever is the wrong answer for any failure class.

Fix

finalizeRemoval now bounds the retry loop. When the cap is hit, the runner logs a warning, frees the local state, and lets the IdleCount slot recycle. The orphaned VM is left for an external sweep, which is what the operator was getting anyway after the retry storm started.

The cap is set well above the existing removeRetryTries threshold that already powers the stuck-on-removing metric, so the metric continues to fire first and operators see the warning sign before the runner gives up.

Testing

Existing tests pass. There is a new regression test that pins the bound by using a permanently-failing remove, asserts finalizeRemoval returns within a small wall-clock window, and asserts the local state is cleared.

Also validated end-to-end in a sandbox runner-manager fleet against three synthetic failure modes injected at the docker-machine layer. Without the cap, machines accumulated in stuck-on-removing and locally-tracked state diverged from GCP. With the cap, neither happened.

References

Merge request reports