Add Replace to swap an unusable acquisition's instance

What does this MR do?

Adds Replace(ctx, key) to Taskscaler. It swaps the instance behind an existing acquisition for a different one, for when the acquired instance turns out to be unusable (for example, the cloud provider terminated it). Also stops Acquire from handing out instances that have reached failure_threshold.

  • Replace: in one step under ts.mu, it relinquishes the old slot and turns the key's claim back into a reservation, so the key never loses its place. It then waits like Acquire, skipping the released instance, and counts as pending demand so the fleet scales up if needed. If no other instance has capacity and taskscaler can't scale up, it falls back to the released instance instead of waiting until ctx expires. It returns ErrAcquisitionNotFound for an unknown key and ErrAcquisitionWasSuspended for a suspended slot.
  • canAcquire skips unhealthy instances (separate commit). Today HealthFailure only increments a counter. An instance past failure_threshold stays acquirable until a scale-down pass removes it, which doesn't match the WithFailureThreshold doc ("failures before an instance is replaced").

Why was this MR needed?

When a runner job can't reach its instance, the runner wants to retry it on a different one. With the current API it can only Release + Reserve + Acquire. That breaks in two ways:

  1. Acquire walks instances in the same order and ignores health, so the retry gets the same dead instance back. Every retry fails on it.
  2. Release and Reserve are separate locked calls. In between, another job or reservation throttling can take the capacity, and an already-accepted job fails with no capacity.

The runner can't fix either from outside: it can't exclude an instance, hold ts.mu across both steps, or add pending demand.

Requested in review of gitlab-org/gitlab-runner!7491 (gitlab-org/gitlab-runner!7491 (comment 3962726590)).

What's the best way to test this MR?

go test -race ./... -run 'Replace|Unhealthy'

The tests cover:

  • the job lands on a different instance;
  • fallback to the same instance when it's the only possible capacity, with and without reservations;
  • no fallback when scale-up is possible;
  • an unknown key;
  • a suspended slot;
  • ctx cancellation;
  • unhealthy instances being skipped by Acquire.

End to end: gitlab-org/gitlab-runner!7491 uses Replace. Its prepare_retry_test.go drives the real taskscaler over the dummy plugin and checks that a job whose first instance is unreachable, or is removed mid-dial, gets prepared on another instance.

Notes for reviewers

  • hasCapacityFromOtherSources mirrors capacity() without the reservation-throttling early return, because Replace isn't subject to throttling. Happy to fold this into capacity() behind a parameter instead.
  • On failure (for example ctx cancelled), Replace has already released the old acquisition and drops the key's reservation. Callers must reserve again before calling Acquire.
  • The canAcquire change means an unhealthy instance can't recover through an acquire-time heartbeat. It's handed out again only after a HealthSuccess from elsewhere, otherwise it waits for removal. It's a separate commit so it can be dropped if that's too strict.

What are the relevant issue numbers?

Related to gitlab-org/gitlab-runner#39145

Edited by Ashvin Sharma

Merge request reports

Loading
Loading