(Instance 6/6) Expire suspended instance executor environments

Ashvin Sharmarequested to merge
feat/suspendable-ttl-instance into feat/suspendable-ttl-docker-sweep

What does this MR do?

TTL for suspended instance executor environments:

  • Bumps taskscaler so Resume refuses a suspended slot past its stamped expiry.
  • Maps a lost or expired autoscaler slot to runtime_environment_unavailable.
  • Requires a dedicated instance to suspend: runner-level FF_SUSPENDABLE_ENVIRONMENTS, a runner ID, capacity_per_instance = 1, max_use_count = 1, use_common_build_dir = true, and no vm_isolation.
  • Reuses the resumed acquisition when prepare is retried.

Why was this MR needed?

The instance executor suspends the whole VM, so its expiry lives on the taskscaler slot. A resume past it must fail without starting the VM, and the sweep must delete the stopped VM.

What's the best way to test this MR?

  • go test -race ./executors/internal/autoscaler/ ./executors/instance/
  • e2e on GCP with backbench: the slot ExpiresAt equals prepare time + TTL. A resume past it fails as runtime_environment_unavailable (suspended slot expired), and GCE's operation log shows no start. After expiry + grace the sweep deletes the stopped VM without starting it, and a later resume is unavailable. A resume within a 10m stamp succeeds after the config TTL is reloaded to 60s.

Stacked on !7667. Depends on gitlab-org/fleeting/taskscaler!118 (merged) (pinned by pseudo-version 70dd8ad); re-pin to a released version once it merges.

What are the relevant issue numbers?

Related to #39468.

Merge request reports