Add Replace to swap an unusable acquisition's instance
What does this MR do?
Adds Replace(ctx, key) to Taskscaler. It swaps the instance behind an existing acquisition for a different one, for when the acquired instance turns out to be unusable (for example, the cloud provider terminated it). Also stops Acquire from handing out instances that have reached failure_threshold.
Replace: in one step underts.mu, it relinquishes the old slot and turns the key's claim back into a reservation, so the key never loses its place. It then waits likeAcquire, skipping the released instance, and counts as pending demand so the fleet scales up if needed. If no other instance has capacity and taskscaler can't scale up, it falls back to the released instance instead of waiting untilctxexpires. It returnsErrAcquisitionNotFoundfor an unknown key andErrAcquisitionWasSuspendedfor a suspended slot.canAcquireskips unhealthy instances (separate commit). TodayHealthFailureonly increments a counter. An instance pastfailure_thresholdstays acquirable until a scale-down pass removes it, which doesn't match theWithFailureThresholddoc ("failures before an instance is replaced").
Why was this MR needed?
When a runner job can't reach its instance, the runner wants to retry it on a different one. With the current API it can only Release + Reserve + Acquire. That breaks in two ways:
Acquirewalks instances in the same order and ignores health, so the retry gets the same dead instance back. Every retry fails on it.ReleaseandReserveare separate locked calls. In between, another job or reservation throttling can take the capacity, and an already-accepted job fails withno capacity.
The runner can't fix either from outside: it can't exclude an instance, hold ts.mu across both steps, or add pending demand.
Requested in review of gitlab-org/gitlab-runner!7491 (gitlab-org/gitlab-runner!7491 (comment 3962726590)).
What's the best way to test this MR?
go test -race ./... -run 'Replace|Unhealthy'
The tests cover:
- the job lands on a different instance;
- fallback to the same instance when it's the only possible capacity, with and without reservations;
- no fallback when scale-up is possible;
- an unknown key;
- a suspended slot;
ctxcancellation;- unhealthy instances being skipped by
Acquire.
End to end: gitlab-org/gitlab-runner!7491 uses Replace. Its prepare_retry_test.go drives the real taskscaler over the dummy plugin and checks that a job whose first instance is unreachable, or is removed mid-dial, gets prepared on another instance.
Notes for reviewers
hasCapacityFromOtherSourcesmirrorscapacity()without the reservation-throttling early return, becauseReplaceisn't subject to throttling. Happy to fold this intocapacity()behind a parameter instead.- On failure (for example
ctxcancelled),Replacehas already released the old acquisition and drops the key's reservation. Callers must reserve again before callingAcquire. - The
canAcquirechange means an unhealthy instance can't recover through an acquire-time heartbeat. It's handed out again only after aHealthSuccessfrom elsewhere, otherwise it waits for removal. It's a separate commit so it can be dropped if that's too strict.
What are the relevant issue numbers?
Related to gitlab-org/gitlab-runner#39145