Token-exchange JWKS keys are fetched once at startup and never refreshed

What

internal/auth/tokenexchange/verifier.go fetches the issuer JWKS once at startup (one-shot, retried until the first success) and never refreshes for the process lifetime. So jwks_last_success_timestamp_seconds is effectively the pod start time (~8h on current prod pods), and jwks_fetches_total stops at the first success.

Why it matters

If the issuer (GitLab cloud-connector) rotates its signing keys, a running pod keeps the old keys and rejects tokens signed with the new ones (auth_failure_reason="invalid_token") until it restarts. After startup AR is not fetching, so there is no JWKS fetch-failure signal, only the downstream validation failures.

Mitigation today

Pod churn refreshes keys indirectly: deploys and autoscaling cycle pods regularly, and each new pod fetches current keys at boot. So in a busy prod setup with regular rollouts and scaling this is unlikely to bite. The residual risk is a long-lived pod that survives across a key rotation without a restart (a low-traffic window with no deploy or scale event).

Options

  1. Add periodic JWKS refresh, or re-fetch on an invalid_token verification failure, so keys track rotation without a restart.
  2. Accept pod churn as the de-facto refresh, document it, and rely on an invalid_token alert as the safety net.

Detection

A sustained auth_client_authentications_total{auth_outcome="failure", auth_failure_reason="invalid_token"} rate flags this and other token-validation breakage. Alerting on it is tracked with the #354 (closed) alert set.

Related to #354 (closed).