# :warning: Summary Two authentication-observability gaps, measured on staging against the `mimir-runway` tenant during the AuthN validation run ([#1225](https://gitlab.com/gitlab-org/ops/artifact-registry/-/work_items/1225), rows 155, 156, 162). ## 1. No metric counts authentication outcomes Of the 193 `gitlab_artifact_registry_*` series on the scrape, none counts client authentication outcomes: there is no `401` counter and no per-reason breakdown (`invalid_token`, `expired`, `keys_unavailable`, `unknown_origin`). The only `token_exchange` metrics are `oci_remote_*`, which are the Artifact Registry authenticating *to* an upstream registry, not clients authenticating to the Artifact Registry. In a four-hour window on staging, 74 `auth: token verification failed` records reached the logs and were counted by no metric. ## 2. A `401` from the auth middleware carries no usable `route` label The authentication middleware rejects a request before it reaches the mux, so `gitlab_artifact_registry_http_requests_total{status_code="401"}` is emitted with an empty `route` label and every format arm's failures collapse into one series. The service-token edge is the only `401` that keeps a route, because its guard is mounted on the mux. An operator cannot tell a Maven `401` from an npm or a container one. ## 3. No JWKS-fetch metric Key-fetch success, failure, and staleness appear only as log lines (`token-exchange: JWKS fetched; verifier ready`). No counter or gauge exists, so no alert can fire on a verifier that never becomes ready — the failure mode [gitlab#616174](https://gitlab.com/gitlab-org/gitlab/-/work_items/616174) describes. # :bar_chart: Evidence Measured 2026-09-08 through the `mimir-runway` tenant and the `clickhouse-runway-production` log datasource (`ServiceName = 'artifact-registry-gke'`). - 193 `gitlab_artifact_registry_*` metric names on the scrape; no `authn`, `token_validation`, or client-facing `token_exchange` counter among them. - `sum by (route) (gitlab_artifact_registry_http_requests_total{status_code="401"})` returns an empty-route bucket carrying the format-arm failures, plus one `/api/gitlab/v1/` series for the mux-mounted service-token edge. - `{__name__=~"gitlab_artifact_registry.*(jwks|verifier|signing).*"}` returns nothing. # :white_check_mark: What is not broken A single authentication failure is fully diagnosable from logs: the `401` warn carries a bounded reason (never a claim value), the request's `correlation_id` joins the client error to its record, and the record carries a populated `trace_id` and `span_id`. The gap is aggregate visibility and alerting, not per-request traceability. # :hammer: Suggested shape A `denials_total`-style counter for authentication outcomes, labelled by a bounded reason and by the surface (format arm) the request targeted, plus a JWKS signal to back an alert. The surface label is the part the empty `route` cannot supply today. ## Implementation notes (from code, at 0c87cef79365) - Scope is every surface, not just container: all verdicts flow through four handler choke points — `auth.Middleware` (five dispatch arms: [table](https://gitlab.com/gitlab-org/ops/artifact-registry/-/blob/0c87cef79365/internal/auth/dispatch.go#L155-216)), `TokenHandler` (`/v2/auth/token`), and the two `servicetoken` middlewares (GitLab API). The three 401 writers behind them are the exhaustive emitter set, so coverage is complete by construction. - A closed reason set is enumerable today: the extraction failures (`missing_credential`, `malformed_header`, `wrong_scheme`, `empty_credential`, `basic_undecodable`, `basic_no_password`, `unsupported_scheme`), `bootstrap_token_mismatch`, the JWT classes from [`mapVerifyError`](https://gitlab.com/gitlab-org/ops/artifact-registry/-/blob/0c87cef79365/internal/auth/tokenexchange/errors.go#L26-39) (`keys_unavailable`, `expired`, `invalid_token`, `unknown_origin`), and the service-token classes (`service_token_mismatch`, `mode_mismatch_*`). Bad-signature, wrong-audience, malformed, unknown-issuer, and bad-version are not separable today — upstream `userauth` collapses all five into `invalid_token`; splitting them is an upstream change, so budget the set at what exists. - Cardinality: the label cannot be a bare `reason` (the audit keys closed sets by label name globally) and cannot contain `token`/`authorization` (banned substrings), so something like `auth_failure_reason`, with rows in both `expectedDistinctValues` and `closedSetValues` in `internal/metrics/cardinality.go` — the `denial_reason` precedent. - Do not count the four client-disconnect returns that write no response (`handler_token.go` names the trap: transport cancellations inflating credential-failure metrics). Anonymous probes are deliberately unlogged today, so a counter sees more than the logs do. - JWKS: the fetch is one-shot at startup (no refresh — the fetch goroutine exits on first success, and key rotation at the issuer requires a restart), and readiness already gates on it. `keys_unavailable` can therefore only occur in the startup window, so the alertable signal is a verifier that never becomes ready (fetch-attempt counter or last-success-age gauge) rather than a steady-state staleness metric. # :books: Precedent The authorization validation run filed the analogous authorization-side observability gaps as [#1141](https://gitlab.com/gitlab-org/ops/artifact-registry/-/work_items/1141), [#1142](https://gitlab.com/gitlab-org/ops/artifact-registry/-/work_items/1142), and [#1143](https://gitlab.com/gitlab-org/ops/artifact-registry/-/work_items/1143). Related to https://gitlab.com/gitlab-org/ops/artifact-registry/-/work_items/1225