Fix jobs stuck in canceling due to missing final update

What does this MR do?

Fix jobs that get stuck in canceling state when GitLab cancels a build before the runner has started the build lifecycle.

When the initial PUT /api/v4/jobs/:id (sent right after job pickup, with state=running) returns Job-Status: canceling, the runner currently calls trace.Finish(), which by design does not send any final state to the server. The build then sits in canceling until Ci::TimedOutBuilds::DropCancelingService or Ci::StuckBuilds::DropCancelingService cleans it up — typically 15 minutes after the job timeout, or 1 hour of idle.

This MR splits the combined UpdateAbort || CancelRequested branch in commands/multi.go ~ requestJob and commands/single.go ~ processBuild:

• UpdateAbort keeps trace.Finish(). The server already considers the job finalised (403/404), so any further state PUT would be rejected.
• CancelRequested now calls trace.Fail(common.ErrJobCanceled, common.JobFailureData{Reason: common.JobCanceled}), which sends PUT state=failed with failure_reason=job_canceled. The server transitions the build from canceling to canceled via the drop event in commit_status.rb, immediately and without waiting for a cleanup cron.

The clientJobTrace is already in Running state at that point (because network.GitLabClient.ProcessJob calls trace.start() before returning), so the state guard in complete() does not block the final update.

The chosen failure reason and error mirror what setTraceStatus does in common/build.go for cancels that arrive mid-execution, keeping the runner-side handling of cancels consistent across both paths.

Why was this MR needed?

Closes #39455.

Reproduced reliably with daniel-prause/race-condition-project on GitLab.com using GitLab Runner 18.11.3, shell executor, darwin/arm64. The "Verification" section below has the details.

This is the same end-state symptom as #37900 (still open) and #37780 (closed) (closed), but a different trigger:

• #37900 / #37780 (closed) describe runner death during cancel (kubernetes pod kill, FF_USE_POD_ACTIVE_DEADLINE_SECONDS, restart). The runner process is gone, so it cannot send a final state.
• This MR addresses the case where the runner is fully alive but the cancel arrives synchronously on the response to the very first UpdateJob call, before the build lifecycle starts. The runner sees the signal and silently drops it.

What's the best way to test this MR?

The fix has tests at the unit level in commands/multi_test.go and commands/single_test.go:

• Existing TestRun*Command_*_HandlesCancelRequested tests have been updated to expect trace.Fail instead of trace.Finish.
• Existing TestRun*Command_*_HandlesUpdateAbort tests are left unchanged, asserting that the abort path still uses trace.Finish.
• New regression tests TestRun*Command_*_CancelRequestedReportsFinalState document the desired behaviour with reference to #39455 in the test docstring.
• New regression tests TestRun*Command_*_UpdateAbortStillFinishesWithoutFinalState guard the abort path against accidental refactors.

go test -race -count=1 -timeout 5m ./commands/... \
  -run "TestRunCommand_requestJob|TestRunSingleCommand_processBuild"

For end-to-end verification, see the next section.

Verification

Reproduced the bug locally against daniel-prause/race-condition-project on GitLab.com.

Without the fix:

Checking for jobs... received      job=14467824555  (POST /jobs/request, 201)
Updating job...                    job=14467824555  bytesize=0
Submitting job to coordinator...ok job=14467824555  job-status=canceling  code=200
                                                    (no further runner activity)

Ci::Build#status stays at canceling until Ci::TimedOutBuilds::DropCancelingService runs after started_at + job_timeout + 15min.

With the fix:

Checking for jobs... received      job=14467824555
Updating job...                    job=14467824555  bytesize=0
Submitting job to coordinator...ok job=14467824555  job-status=canceling  code=200
Updating job...                    job=14467824555                          (final-state PUT)
Submitting job to coordinator...ok job=14467824555                          (server accepts)

Ci::Build#status transitions to canceled within seconds. Verified via the rails console.

Root cause

When a job is picked up via POST /jobs/request, the runner immediately follows up with an initial PUT /api/v4/jobs/:id setting state=running. This happens in commands/multi.go ~ requestJob (and the equivalent path in commands/single.go ~ processBuild), before Build.Run is invoked.

If the GitLab server returns Job-Status: canceling on that initial PUT, the runner currently takes this branch:

if updateResult.State == common.UpdateAbort || updateResult.CancelRequested {
    trace.Finish()
    return nil, nil, retried, nil
}

trace.Finish() is documented as local-only cleanup, with no API call:

// Finish cleans up the trace without sending updates. Either Success, Fail or Finish must be called
// and only once.
func (c *clientJobTrace) Finish() {
    c.buffer.Finish()
    c.finished <- true
    c.buffer.Close()
}

The server-side state machine in commit_status.rb defines canceling → canceled only via the success and drop events:

event :success do
  transition canceling: :canceled  # runner returns success/failed
end

event :drop do
  transition canceling: :canceled  # runner returns success/failed
end

So GitLab sets the build to canceling and waits for the runner to confirm with a final state. When the runner uses trace.Finish() instead, the build is never finalised. Conflating UpdateAbort (server already finalised the job) and CancelRequested (server is waiting for the runner to finalise) is the root cause: they have different semantics and require different runner-side responses.

Author's checklist

• Followed commit message guidelines
• Added tests to confirm the change works as expected
• Added/updated documentation if needed (no user-facing behaviour change beyond fixing the hang)
• If this change affects the user experience, added/updated screenshots (n/a — internal API behaviour)
• Locally ran make build_simple and verified the change
• Followed the style guides
• For the GitLab project: added the relevant changelog category to the commit
• If this MR fully solves the issue, used the Closes #39455 keyword

Related issues

• Closes #39455
• Related: #37900 (open, kubernetes runner restart during canceling — same end state, different trigger)
• Related: #37780 (closed) (closed, kubernetes runner pod kill via FF_USE_POD_ACTIVE_DEADLINE_SECONDS)
• Server-side context: app/models/commit_status.rb state machine in gitlab-org/gitlab

Notes for reviewers

• The multi-runner test required threading a non-nil configfile into the RunCommand mock because mr.log() accesses mr.configfile.Config(). This matches how resetRunnerTokenTestController initialises RunCommand for tests elsewhere in the file.
• Rollout: safe to ship without a feature flag. The behaviour being replaced (Finish() with no final state) is itself a silent protocol violation, so the change is strictly an improvement; reverting is also safe in case of unexpected regressions.