# :mag: Summary `gitlab_artifact_registry_authz_denials_total` carries two `denial_reason` values that cannot fire, and excludes the denial that actually happens. Measured on staging over 24 hours across 9 pods, covering a 99-scenario authorization validation run: | `denial_reason` | Occurrences | | --- | --- | | `no_role_assignment` | 160 | | `unparsed_route` | 15 | | `unresolved_repository` | 6 | | `escaped_separator` | 3 (reproduced deliberately) | | `glaz_denied` | **0** | | `anonymous` | **0** | | `unresolved_principal` | 0 (correct — it marks an IAM contract violation) | Meanwhile 42 requests in the same window were denied by GLAZ for the caller's access level, logged as `authz: glaz check decision`, and counted nowhere. # :bug: The three findings ## 1. The permission-boundary denial is counted nowhere A principal refused an operation their role does not permit answers `403` and increments no counter. `denials_total` is the only authorization counter, so the commonest refusal in normal operation is chartable only from logs. The 42 occurrences break down as: | Action | Count | | --- | --- | | `create_artifact` | 18 | | `delete_artifact` | 12 | | `update_repository` | 5 | | `delete_repository` | 4 | | `create_repository` | 3 | ## 2. `glaz_denied` cannot fire Both increment sites in `denialMapping` (`internal/authz/decision.go`) require a principal that **holds a tuple but cannot read**: - the read arm needs a denied `read_artifact`/`read_repository` with tuples present - the non-read arm needs a denied write whose follow-up read Check **also** denies Every role in ADR-021's permission table grants `read_artifact` and `read_repository`, so any principal holding a tuple can read: the follow-up Check allows and the request becomes the uncounted `403` above. A principal holding no tuple short-circuits to `no_role_assignment` before GLAZ is consulted. So the label named for the GLAZ-denied case never fires, while the real GLAZ-denied case is excluded. `gitlab_artifact_registry_glaz_requests_total` shows 300 `rpc="Check"` completions in the same window, so GLAZ is being consulted normally. ## 3. `anonymous` cannot fire `AnonymousDenied` increments it, but a request carrying no `Authorization` header is refused `401` by token verification first. Verified on all four surfaces. # :arrows_counterclockwise: Steps to reproduce Prerequisites: a namespace with a hosted repository, and a principal holding `ARTIFACT_VIEWER` on it. `$JWT` is that principal's token-exchange JWT, `$HOST` the registry host, `$SLUG` the namespace slug. ## Finding 1 and 2 — a refused write is logged and not counted 1. Sample the counter: ```promql sum by (denial_reason) (gitlab_artifact_registry_authz_denials_total{env="staging"}) ``` 1. Attempt a write the role does not permit: ```shell curl -i -X PATCH \ -H "Authorization: Bearer $JWT" \ -H 'Content-Type: application/json' \ -d '{"description":"probe"}' \ "https://$HOST/api/v1/$SLUG/repositories/$REPOSITORY" ``` Answers `403` with `{"error":{"code":"forbidden","message":"access denied"}}`. 1. Confirm the decision was logged, keyed on the `x-request-id` the response carried: ```sql SELECT JSONExtractString(Body,'msg') AS msg, JSONExtractString(Body,'action') AS action, JSONExtractString(Body,'decision_reason') AS reason FROM observability.otel_logs WHERE ServiceName = 'artifact-registry-gke' AND Attributes['env'] = 'staging' AND JSONExtractString(Body,'correlation_id') = '<x-request-id>' ``` Returns `authz: glaz check decision`, `update_repository`, `denied: no matching policy`. 1. Re-sample the counter. **No value changed**, and `glaz_denied` has no series. ## Finding 2 — the states that would reach `glaz_denied` all land elsewhere Each of these was attempted; none moved `glaz_denied`: | Attempt | Result | | --- | --- | | Namespace-`ARTIFACT_VIEWER` principal, also `ARTIFACT_CONTRIBUTOR` on other repositories, `PATCH` on a repository where it holds only the inherited read | `403` | | `ARTIFACT_VIEWER` principal, `DELETE` a package | `403` | | Principal with repository grants but no namespace role, `GET /api/v1/$SLUG/statistics` | `404`, booked `no_role_assignment` | ## Finding 3 — a credential-less request never reaches `AnonymousDenied` Each of these answers `401`, and `anonymous` stays absent: ```shell curl -o /dev/null -w '%{http_code}\n' "https://$HOST/v2/" curl -o /dev/null -w '%{http_code}\n' "https://$HOST/v2/$SLUG/container/$REPOSITORY/$IMAGE/manifests/$TAG" curl -o /dev/null -w '%{http_code}\n' "https://$HOST/$SLUG/npm/$REPOSITORY/$PACKAGE" curl -o /dev/null -w '%{http_code}\n' "https://$HOST/$SLUG/maven/$REPOSITORY/$PATH" ``` To see historical values across pods that have since been replaced, a counter is per process, so aggregate rather than reading the instant value: ```promql max by (denial_reason, pod) (max_over_time(gitlab_artifact_registry_authz_denials_total{env="staging"}[24h])) ``` # :dart: Expected An operator can chart and alert on authorization refusals. Either the `403` arm books `glaz_denied`, which matches both the label's name and the log line already emitted there, or it books a new reason and `glaz_denied` is retired. `anonymous` either becomes reachable or is removed. The catalog entry in `docs/dev/observability.md` states the exclusion of `403` and `503` deliberately, so the counter is honest about itself. The defect is that nothing else covers that class, and that two of the documented values are unreachable. # :bulb: Why this matters now [#354](https://gitlab.com/gitlab-org/ops/artifact-registry/-/work_items/354) builds the service's dashboards and alerting on these metrics. An alert written against `denials_total` today cannot see the permission-boundary denial at all, and an alert written against `glaz_denied` can never fire. Found during the staging authorization validation recorded in [#1067](https://gitlab.com/gitlab-org/ops/artifact-registry/-/work_items/1067), observability rows 106 to 108.