Add an auth probe to the boot-verify canary

What does this MR do?

Adds an opt-in verify_auth check to the boot-verify canary. When enabled, the runner manager calls POST runners/verify before running the synthetic job, with the same token and the same Runner-Token header construction as the job request poll. Any failure fails the canary, whether the server rejected the token or the request could not be completed at all (DNS, routing, TLS, or a token that is not a valid header value). The manager then never reports Ready and the previous manager keeps running.

[runners.experimental.boot_verify]
  enabled = true
  verify_auth = true   # new, default false

The check is single-shot, like the other probes. A failed canary exits the process non-zero and the orchestrator restarts it, so a transient network failure costs one restart, while a bad token keeps the pod in a restart loop that never becomes Ready.

To support this, Network.VerifyRunner now returns (*VerifyRunnerResponse, error) instead of only a response pointer. Previously it returned a non-nil response on transport and server failures, so callers could not tell "token accepted" from "request never completed". The new contract: a response on 200, (nil, nil) on 403, and an error otherwise. The verify command behaves as before and only deletes runners when the server rejected the token. One intentional change in register: registering with a runner token now aborts on a network failure instead of proceeding unverified, matching its existing "You may be having network problems" error message.

Why not probe jobs/request directly?

The failing call in production was POST /api/v4/jobs/request, but an authenticated request there can be assigned a real pending job (old and new managers share the token during a rollout), and an assigned job cannot be declined, so the canary would fail a user's job or leave it stuck. runners/verify authenticates the same token with the same header against the same base URL (on GitLab.com fleets, the ci-gateway) with no side effects, and catches both failure modes from the incident.

Why was this MR needed?

During a rollout, a runner manager booted with a runner token containing an invalid header character. The canary's synthetic job never touches the GitLab API, so boot-verify passed, the manager reported Ready, and the old manager was shut down. Every subsequent job request then failed client-side:

Checking for jobs... failed ... net/http: invalid header field value for "Runner-Token"

The manager was Ready but could never fetch a job. The same gap applies to a revoked or rotated token that the server rejects.

What's the best way to test this MR?

Unit tests cover the probe (pass, rejection, request failure) and the changed VerifyRunner contract. For end-to-end validation: enable verify_auth on a runner with a valid token and confirm the canary passes, then corrupt the token (append %0A or use a revoked token) and confirm the canary fails and the manager never reports Ready.

The repo's committed mocks predate the mockery version pinned in the Makefile, and regenerating rewrites every mock file with unrelated interface{} to any churn, so the MockNetwork.VerifyRunner block was updated by hand in the existing style. A repo-wide regeneration should be a separate MR.

What are the relevant issue numbers?

Related MRs: !7214 (merged), !7265 (merged).

Edited by Igor

Merge request reports

Loading
Loading