Keep suspended instances resumable across a restart
What does this MR do?
Keeps a suspended environment resumable when Runner restarts, with state_storage enabled:
- Restart while the VM is still stopping: an instance first seen
Suspending/Resumingis adopted with its saved state, instead of the state being dropped (failed to restore saved state). - Resume wait: a resume stops waiting as soon as its instance is removed, instead of waiting out
instance_acquire_timeout. - Unclaimed suspended instances: a suspended instance that holds no slot, so nothing can resume it, is removed
idle_timeafter the restarted Runner first sees it (or on the first pass whenidle_timeisn't set), instead of staying stopped indefinitely.
The code and tests are Arran's prototype from gitlab-org/gitlab-runner#40016, rebased onto main. The restored-flag fix from that prototype has already landed in !116 (merged), so it is not repeated here.
Why was this MR needed?
The state_storage path is the only one Runner allows for suspended environments, and a Runner restart is routine. Without this change, a restart that lands while the suspended VM is still stopping loses the environment and leaves the stopped VM behind.
Unclaimed instances are removed on idle_time rather than the TTL. Runner requires state_storage for suspend, so a suspended instance with no slot can never be resumed, and removing it early never exceeds the TTL.
What's the best way to test this MR?
go test -race ./.... New tests are the TestRestartResume_* tests in taskscaler_test.go. Five of them fail on main (RestartWhileSuspendingKeepsState, RestartWhileResumingKeepsState, RestartWhileResumingWithoutStateIsAcquirable, RemovedInstanceEndsWait, UnclaimedSuspendedInstanceRemoved).
End to end on GCP (docker-autoscaler, fleeting-plugin-googlecloud, backbench), Runner built from the top of the suspendable-TTL stack, with taskscaler with and without this change:
| Runner restart | Without this MR | With this MR |
|---|---|---|
| Graceful (SIGQUIT), VM already stopped | ||
| Crash (SIGKILL), VM already stopped | ||
| Restart while the VM is still stopping | failed to restore saved state, then acquisition not found; stopped VM left behind |
boot_id changed |
The first two rows already pass thanks to !116 (merged). The resume wait and unclaimed-instance removal are covered by the unit tests only.
What are the relevant issue numbers?
Related to gitlab-org/gitlab-runner#40016