Auth: OAuth token fetch failure during /auth callback serves 503

Background

While reviewing !1280 (merged) (recover from 403 on expired SSO session by re-authenticating), it was noted that the 503 symptoms reported in gitlab-com/request-for-help#4770 come from a different code path that the MR deliberately does not touch.

Problem

In internal/auth/auth.go, checkAuthenticationResponse serves a 503 (httperrors.Serve503) when the OAuth access-token fetch (fetchAccessToken) fails during the /auth callback:

The user has just completed the OAuth round-trip, so a transient failure here drops them on a bare 503 page with no retry path other than reloading and re-running the whole flow.
Combined with SSO session expiry (now handled by !1280 (merged)), this was a contributor to the intermittent errors customers reported.

Proposal

Investigate and improve resilience of the token fetch during the callback, for example:

Retry transient failures of the token endpoint before serving an error.
Serve a friendlier error page that links back to the originally requested URI so the OAuth flow can restart cleanly.
Add logging/metrics to distinguish token-endpoint failures from other 503 sources.

Decisions (revised 2026-10-22, as implemented by the MR stack)

<🤖>

Fable 5.1, acting for @armin.pasalic. Supersedes the 2026-08-18 decisions, kept collapsed below for the record.

What holds: a process-level fix with no new config flags, the OAuth error/error_description body logged, an alertable counter (gitlab_pages_auth_token_fetch_failures_total{reason,outcome}, plus gitlab_pages_auth_token_fetch_recoveries_total{via}), and no retry loop.

What changed:

The token exchange is retried once in place, after 250 ms, for transport errors other than timeouts, upstream 5xx and invalid_grant (!1336 (closed)). A spent code fails that retry within milliseconds; a grant not yet visible on a lagging replica, or a dropped connection, recovers without a browser round trip.
The restart with a fresh code is gated by failure class and bounded to once a minute per host, with the marker in a dedicated gitlab-pages-auth-recovery cookie rather than the session (!1337 (closed)). Rate limits, configuration errors and malformed responses go straight to the terminal response.
The "friendlier error page" is no longer deferred. A terminal failure is shown once, as a 503 at the requested page with a "Try again" link, so a reload starts a clean sign-in instead of replaying a spent code (!1338 (closed)).
Retry-After from a 429 is used in the page copy, not sent as a header.

Why the in-place retry came back

Doorkeeper's validate_grant fails the same way for a missing grant and a revoked one, so a spent code and a grant that a lagging replica has not seen yet both come back as invalid_grant; and Pages' server-to-server POST /oauth/token carries no user session, so GitLab's load-balancer sticking does not apply to it. GitLab has since merged gitlab!257390 (merged), rolling out behind stick_new_oauth_tokens_to_primary (gitlab#632580 (closed)) and oauth_lookups_use_load_balancer_sticking (gitlab#632808 (closed)). Once both are on, the lag case should largely disappear and the retry remains as a safety net for dropped connections and upstream 5xx.

</🤖>

Decisions (refined 2026-08-18), superseded

Scope narrowed to process-level retry + observability:

Restart the OAuth flow once on token-fetch failure (fresh authorization code) rather than retrying the spent code in place. Codes are single-use, so invalid_grant — the likely dominant failure — cannot be fixed by an in-place retry.
One-shot session marker guards the restart, mirroring sso_reauth_attempted from !1280 (merged). Second failure still serves 503.
Log the OAuth error/error_description body (discarded today) and add an alertable counter to separate token-endpoint failures from other 503s.
No new config flags: auth-timeout already bounds the fetch.
Deferred: the "friendlier error page" bullet under Proposal above.
Dropped: Retry-After, as browsers do not auto-retry 503 navigations.

Implementation must follow double-loop TDD with minimal code comments. See the Workplan for the full breakdown.

Merge requests

<🤖>

Three MRs cut from the final design on 2026-10-23, after the review of the earlier six-MR stack; each targets the previous one's branch and retargets automatically as it merges:

!1334 (closed) contract-faithful OAuth mode for the GitLab test stub (test only, targets master)
!1370 (closed) typed token-fetch errors, one in-place retry and the counters; a terminal failure still serves 503 at this step
!1371 (closed) restart with a fresh code, bounded to once a minute per host, and the failure page at the requested page; closes this issue

The earlier stack (!1309 (closed), !1334 (closed) in its old form, !1335 (closed), !1336 (closed), !1337 (closed), !1338 (closed)) carried intermediate designs that later MRs replaced; the superseded MRs are closed with a pointer.