Keep suspended instances resumable across a restart

What does this MR do?

Keeps a suspended environment resumable when Runner restarts, with state_storage enabled:

  • Restart while the VM is still stopping: an instance first seen Suspending/Resuming is adopted with its saved state, instead of the state being dropped (failed to restore saved state).
  • Resume wait: a resume stops waiting as soon as its instance is removed, instead of waiting out instance_acquire_timeout.
  • Unclaimed suspended instances: a suspended instance that holds no slot, so nothing can resume it, is removed idle_time after the restarted Runner first sees it (or on the first pass when idle_time isn't set), instead of staying stopped indefinitely.

The code and tests are Arran's prototype from gitlab-org/gitlab-runner#40016, rebased onto main. The restored-flag fix from that prototype has already landed in !116 (merged), so it is not repeated here.

Why was this MR needed?

The state_storage path is the only one Runner allows for suspended environments, and a Runner restart is routine. Without this change, a restart that lands while the suspended VM is still stopping loses the environment and leaves the stopped VM behind.

Unclaimed instances are removed on idle_time rather than the TTL. Runner requires state_storage for suspend, so a suspended instance with no slot can never be resumed, and removing it early never exceeds the TTL.

What's the best way to test this MR?

go test -race ./.... New tests are the TestRestartResume_* tests in taskscaler_test.go. Five of them fail on main (RestartWhileSuspendingKeepsState, RestartWhileResumingKeepsState, RestartWhileResumingWithoutStateIsAcquirable, RemovedInstanceEndsWait, UnclaimedSuspendedInstanceRemoved).

End to end on GCP (docker-autoscaler, fleeting-plugin-googlecloud, backbench), Runner built from the top of the suspendable-TTL stack, with taskscaler with and without this change:

Runner restart Without this MR With this MR
Graceful (SIGQUIT), VM already stopped ✅ ✅
Crash (SIGKILL), VM already stopped ✅ ✅
Restart while the VM is still stopping ❌ failed to restore saved state, then acquisition not found; stopped VM left behind ✅ same VM resumed, marker read back, boot_id changed

The first two rows already pass thanks to !116 (merged). The resume wait and unclaimed-instance removal are covered by the unit tests only.

What are the relevant issue numbers?

Related to gitlab-org/gitlab-runner#40016

Edited by Ashvin Sharma

Merge request reports

Loading
Loading