## Background While reviewing !1280 (recover from 403 on expired SSO session by re-authenticating), it was noted that the 503 symptoms reported in gitlab-com/request-for-help#4770 come from a different code path that the MR deliberately does not touch. ## Problem In `internal/auth/auth.go`, `checkAuthenticationResponse` serves a 503 (`httperrors.Serve503`) when the OAuth access-token fetch (`fetchAccessToken`) fails during the `/auth` callback: - The user has just completed the OAuth round-trip, so a transient failure here drops them on a bare 503 page with no retry path other than reloading and re-running the whole flow. - Combined with SSO session expiry (now handled by !1280), this was a contributor to the intermittent errors customers reported. ## Proposal Investigate and improve resilience of the token fetch during the callback, for example: - Retry transient failures of the token endpoint before serving an error. - Serve a friendlier error page that links back to the originally requested URI so the OAuth flow can restart cleanly. - Add logging/metrics to distinguish token-endpoint failures from other 503 sources. ## Decisions (revised 2026-10-22, as implemented by the MR stack) <:robot:> Fable 5.1, acting for @armin.pasalic. Supersedes the 2026-08-18 decisions, kept collapsed below for the record. What holds: a process-level fix with no new config flags, the OAuth `error`/`error_description` body logged, an alertable counter (`gitlab_pages_auth_token_fetch_failures_total{reason,outcome}`, plus `gitlab_pages_auth_token_fetch_recoveries_total{via}`), and no retry loop. What changed: 1. The token exchange is retried once in place, after 250 ms, for transport errors other than timeouts, upstream 5xx and `invalid_grant` (!1336). A spent code fails that retry within milliseconds; a grant not yet visible on a lagging replica, or a dropped connection, recovers without a browser round trip. 2. The restart with a fresh code is gated by failure class and bounded to once a minute per host, with the marker in a dedicated `gitlab-pages-auth-recovery` cookie rather than the session (!1337). Rate limits, configuration errors and malformed responses go straight to the terminal response. 3. The "friendlier error page" is no longer deferred. A terminal failure is shown once, as a 503 at the requested page with a "Try again" link, so a reload starts a clean sign-in instead of replaying a spent code (!1338). 4. `Retry-After` from a 429 is used in the page copy, not sent as a header. <details> <summary>Why the in-place retry came back</summary> Doorkeeper's `validate_grant` fails the same way for a missing grant and a revoked one, so a spent code and a grant that a lagging replica has not seen yet both come back as `invalid_grant`; and Pages' server-to-server `POST /oauth/token` carries no user session, so GitLab's load-balancer sticking does not apply to it. GitLab has since merged gitlab-org/gitlab!257390, rolling out behind `stick_new_oauth_tokens_to_primary` (gitlab-org/gitlab#632580) and `oauth_lookups_use_load_balancer_sticking` (gitlab-org/gitlab#632808). Once both are on, the lag case should largely disappear and the retry remains as a safety net for dropped connections and upstream 5xx. </details> </:robot:> <details> <summary>Decisions (refined 2026-08-18), superseded</summary> Scope narrowed to **process-level retry + observability**: - Restart the OAuth flow once on token-fetch failure (fresh authorization code) rather than retrying the spent code in place. Codes are single-use, so `invalid_grant` — the likely dominant failure — cannot be fixed by an in-place retry. - One-shot session marker guards the restart, mirroring `sso_reauth_attempted` from !1280. Second failure still serves 503. - Log the OAuth `error`/`error_description` body (discarded today) and add an alertable counter to separate token-endpoint failures from other 503s. - No new config flags: `auth-timeout` already bounds the fetch. - Deferred: the "friendlier error page" bullet under Proposal above. - Dropped: `Retry-After`, as browsers do not auto-retry 503 navigations. Implementation must follow double-loop TDD with minimal code comments. See the Workplan for the full breakdown. </details> ## Merge requests <:robot:> Three MRs cut from the final design on 2026-10-23, after the review of the earlier six-MR stack; each targets the previous one's branch and retargets automatically as it merges: 1. !1334 contract-faithful OAuth mode for the GitLab test stub (test only, targets `master`) 2. !1370 typed token-fetch errors, one in-place retry and the counters; a terminal failure still serves 503 at this step 3. !1371 restart with a fresh code, bounded to once a minute per host, and the failure page at the requested page; closes this issue The earlier stack (!1309, !1334 in its old form, !1335, !1336, !1337, !1338) carried intermediate designs that later MRs replaced; the superseded MRs are closed with a pointer. </:robot:> ## Related - !1280 - gitlab-com/request-for-help#4770 - #1213 (same customer, different code path: a `pages_access` 401 right after login served as a 404) - gitlab-org/gitlab!257390 with gitlab-org/gitlab#632580 and gitlab-org/gitlab#632808 (GitLab side: stick new OAuth grants and tokens to the primary)