feat(auth): count client authentication outcomes and JWKS fetches
What
Two metric families so client-authentication outcomes and JWKS health are visible in
aggregate. Today they exist only as log lines: 74 auth: token verification failed
records in a four-hour staging window were counted by nothing, and a middleware 401
reaches http_requests_total with an empty route, so every format arm's failures
collapse into one series.
| Name | Type | Labels | Where |
|---|---|---|---|
gitlab_artifact_registry_auth_client_authentications_total |
Counter | auth_surface, auth_outcome, auth_failure_reason |
one verdict per request at auth.Middleware, TokenHandler, and both servicetoken middlewares |
gitlab_artifact_registry_auth_jwks_fetches_total |
Counter | result |
the token-exchange verifier's fetch hooks |
gitlab_artifact_registry_auth_jwks_last_success_timestamp_seconds |
Gauge | — | the fetch success hook |
auth_surface comes off the matched dispatch row, which is the part the empty route
label cannot supply. auth_failure_reason is threaded from where each cause is decided
(ExtractCredential, mapVerifyError, the bootstrap comparator, the service-token
branches) through a new auth.WithReason tag; nothing re-derives a class from an error
message, and the tag never reaches the client. The returns that write no response
because the connection is gone book nothing. Anonymous probes are counted, so this
counter sees more 401s than the logs, which leave them out by design.
The JWKS fetch runs at most once per process: it retries until the first success and
then stops, and keys are never refreshed for the process lifetime. That shape decides
the instrumentation. A duration histogram would store a single sample per pod, so
there is none. The gauge holds the one success's timestamp (0 until it happens), and
the counter records the attempts. The alertable signal is a verifier that never
becomes ready — jwks_fetches_total{result="error"} rising while the gauge is still
0 — and the rule for it is written out in docs/dev/alerting.md under Token exchange,
where the label-set join and the for: it needs are stated.
All three label vocabularies are pinned closed in internal/metrics/cardinality.go.
Verified end to end on the branch: the full smoke suite passes (73/73) and the counter
books the expected per-surface successes and failures under it, including the
Bearer-on-Maven rejection as wrong_scheme.
No e2e scenario is added or affected: this is metrics only, with no change to any status code, response body, header, or dispatch decision.
Closes #1267 (closed)