Token-exchange JWKS keys are fetched once at startup and never refreshed
What
internal/auth/tokenexchange/verifier.go fetches the issuer JWKS once at
startup (one-shot, retried until the first success) and never refreshes for
the process lifetime. So jwks_last_success_timestamp_seconds is effectively
the pod start time (~8h on current prod pods), and jwks_fetches_total stops
at the first success.
Why it matters
If the issuer (GitLab cloud-connector) rotates its signing keys, a running pod
keeps the old keys and rejects tokens signed with the new ones
(auth_failure_reason="invalid_token") until it restarts. After startup AR is
not fetching, so there is no JWKS fetch-failure signal, only the downstream
validation failures.
Mitigation today
Pod churn refreshes keys indirectly: deploys and autoscaling cycle pods regularly, and each new pod fetches current keys at boot. So in a busy prod setup with regular rollouts and scaling this is unlikely to bite. The residual risk is a long-lived pod that survives across a key rotation without a restart (a low-traffic window with no deploy or scale event).
Options
- Add periodic JWKS refresh, or re-fetch on an
invalid_tokenverification failure, so keys track rotation without a restart. - Accept pod churn as the de-facto refresh, document it, and rely on an
invalid_tokenalert as the safety net.
Detection
A sustained auth_client_authentications_total{auth_outcome="failure", auth_failure_reason="invalid_token"} rate flags this and other
token-validation breakage. Alerting on it is tracked with the #354 (closed) alert set.
Related to #354 (closed).