## Context An Artifact Registry (AR) release can call an IAM RPC that the deployed IAM does not serve. Nothing before release compares the two versions. The RPC in the report, `LookupResources`, has one call site, the repository listing, which drains it for a caller without a namespace-level role ([S09, Repository listing](https://gitlab.com/gitlab-org/ops/artifact-registry/-/blob/main/docs/specs/S09-authorization.md#repository-listing)). IAM first serves it in v1.49.0. An older IAM answers `Unimplemented`, and the listing then answers the 503 that S09 specifies for any evaluation failure. The [2026-09-30 comment](https://gitlab.com/gitlab-org/ops/artifact-registry/-/work_items/1327#note_3930927578) sets the constraint. Self-Managed and Dedicated take GitLab monthly, and the zero-downtime upgrade policy requires consecutive GitLab minors to work together, so calls between modules have to stay compatible along the monolith's release cycle. This issue serves the Dedicated deployment row (P0) of the GA epic gitlab-org&22707+s. Its tracking issue rules out a behavior change reaching Dedicated without a communication path, because Dedicated cannot toggle feature flags: gitlab-org/gitlab#605684+s. The GitLab-to-AR half of the same question is gitlab-org/ops/artifact-registry#1468+s. ## What exists on main Each line holds at `origin/main` `eca9f1bab` (2026-10-01) unless it names another source. - `internal/iam/client.go`: issues `ReadRelationships` (`:130`), `LookupResources` (`:145`), and `grpc.health.v1.Health/Check` (`:159`), the closed set in `rpcMethods` (`internal/iam/metrics.go:115`). Any `ReadRelationships` error answers 503 (`internal/authz/decision.go:238`). The listing's drain is the one `LookupResources` call site. - `internal/iam/errors.go`, `mapStatus` (`:30`): maps five codes to sentinels. `Unimplemented` takes the default arm (`:53`), a bare `iam: rpc:` wrap, as S25 `### Errors` specifies for any other code. The GLAZ client maps it to `ErrUnimplemented` (`internal/authz/glaz/errors.go:20`), whose comment says the log line then tells a missing RPC from an outage. - `internal/managementapi/list_authz.go`: a refused drain logs a WARN (`:467`) and an ERROR (`:243`), answers 503 (`:245`), and books `listing_failure_reason="transport"` unless its own budget ran out (`:474`). `docs/dev/observability.md:145` defines `transport` as a `LookupResources` call IAM refused or could not answer, so `Unimplemented` books the same value as a network fault. - `internal/grpcutil/metrics.go:84`: each IAM RPC records `gitlab_artifact_registry_iam_data_access_requests_total{rpc, grpc_code}`, so the reported failure counts as `grpc_code="Unimplemented"`. `docs/dev/alerting.md` has no entry on that series. - `cmd/artifact-registry/wire_iam.go:40`: registers no readiness check on IAM, as S25 `## Dependencies` requires. The one `Client.Health` call site is `scripts/iam-rig-wait-healthy`. - `vendir.yml:9`: vendors the IAM proto at v1.24.0 (2026-07-02), which declares `LookupResources` (`third_party/iam-proto/proto/lookup/lookup.proto:33`) and a `HealthResponse` carrying only `status` (`:49`). `buf:generate-check` (`.gitlab-ci.yml:474`) fails only when the committed stubs differ from these protos (`docs/dev/vendored-iam-protos.md`, `## CI gate`). - `gitlab-org/auth/iam`, read 2026-10-01: v1.48.0's `LookupService` defines no `LookupResources` and embeds `UnimplementedLookupServiceServer`, so the call answers `Unimplemented`. [v1.49.0](https://gitlab.com/gitlab-org/auth/iam/-/tags/v1.49.0) (2026-09-04) implements it. From v1.38.0 (2026-08-07), `HealthResponse` carries `version` and `commit` from the build, with the comment "Lets callers confirm a new deploy has actually rolled out before exercising it". - `1cddbec14` (2026-09-10, first in AR v2.30.0) wired the drain into the listing, so AR's floor is IAM v1.49.0. No file states it. Its plan (`docs/plans/2026-09-04-lookup-resources-list-visibility.md`) found that the v1.24.0 proto already carried the messages, "so no `vendir` bump is needed" (`:60`). The plan re-confirmed IAM's behavior by hand (`:274`). - `.gitlab-ci.yml`, `test:integration:iam-rig` (`:1295`): runs `internal/iam`'s integration suite, `LookupResources` included, against `auth/iam:v1.60.2` (`:1303`), the tag Renovate last moved on 2026-09-29 (`f8bdb4977`). It runs on `main` and on canonical MRs that touch its listed paths, never on the security mirror that cuts releases. - `.gitlab/ci/e2e-caproni-functional.gitlab-ci.yml:24`: `test:e2e:caproni:functional` runs the e2e suite on the "Caproni e2e daily" schedule and as a manual, `allow_failure` job on MRs and branches. Its IAM release comes from a vendored fragment that names the chart with no `version` (`test/caproni/vendor/gitlab-caproni/fragments/caproni.iam-data-access.yaml:75`), and nothing under `e2e/` reads an IAM version. The 1.44.0 pin that the original description cites (`test/caproni/fragments/caproni.authz.yaml`) is not on `main`. The authz arm landed on 2026-10-01 without it, through gitlab-org/ops/artifact-registry!2052+s. - `.runway/fairway.yaml:135` and `scripts/chart-gen.sh`: production and the e2e rig take the GLAZ sidecar from one pin, v10.1.0. `vendir.yml` holds two more GLAZ pins: the proto at v6.0.1 (`:64`), whose comment says it tracks the sidecar, and the role catalog at v2.0.0 (`:36`), whose comment says it matches the proto. Only comments tie the three together, and no two pins match. - `docs/dev/releases.md`, `## Deployment is triggered from the mirror's main pipeline`: each release uploads manifests to staging automatically and to production on a manual play. No step reads an IAM version. - `docs/dev/self-managed-install.md`: the header records the one set walked together (GitLab 19.4.0, AR chart 2.82.1, IAM chart 1.55.3, 2026-09-22). `## Scope` records that no matrix exists. `## Upgrading` upgrades AR before IAM, names no required order, and reads IAM's version from `gitlab.iam.update.v1.UpdateService/Health`. - `docs/dev/self-managed-limitations.md`, row 9 of `## What should be lifted`: asks for a published matrix and names this issue for the AR-to-IAM half. - S25 (`docs/specs/S25-iam-relationships-client.md`): `### Proto schema sourcing` makes each proto bump an explicit commit and names no minimum IAM version. `### Connection` points the probe at `grpc.health.v1.Health/Check`, "not the `LookupService.Health` RPC". Acceptance criterion 14's integration test names `ReadRelationships` and `Health` only. - S09 (`docs/specs/S09-authorization.md`), `## Dependencies` (`:38`): states one version requirement, a GLAZ sidecar serving `BatchCheck` (auth/glaz v2.0.0) "confirmed before the consuming changes ship", and none for IAM. - The [release manifest design](https://handbook.gitlab.com/handbook/engineering/architecture/design-documents/release_manifest/) records one compatible version per component for each GitLab release, in a private catalog until that release publishes. Its [schema](https://gitlab.com/gitlab-org/release/manifests/schema/-/blob/main/json-schemas/release-manifest.schema.json) allows only `version`, `sha`, `ref`, and `source` per entry (`additionalProperties: false`), so no entry can state a requirement on another component. The public [record](https://gitlab.com/gitlab-org/release/manifests/released) holds only `19/4/0.json`, which lists `release-platform-canary` alone. IAM's `.gitlab-ci.yml` enables the same `release-platform` component as AR's. ## Change AR records, in one file, its floor: the oldest IAM release that serves every RPC AR calls. A check fails before an AR release reaches an IAM below the floor. At `eca9f1bab` the floor is v1.49.0, set by `LookupResources`. `test:integration:iam-rig` runs at the floor, so an MR that adopts an RPC above it fails until the MR raises the floor in review. An `Unimplemented` reply returns a named error, as it does on the GLAZ client. An alert names the RPC. The listing keeps its 503, which the original description calls correct. Product decides where Self-Managed and Dedicated operators read the floor and which upgrade order they follow. The GitLab-to-AR half stays out of scope, and so does the GLAZ pin drift, which no issue tracks. ## Acceptance criteria 1. Decision for product, since no spec covers it: where Self-Managed and Dedicated operators read AR's IAM floor. The [2026-09-29 comment](https://gitlab.com/gitlab-org/ops/artifact-registry/-/work_items/1327#note_3921922311) points to the release manifest, which pairs co-released versions and has no field for a floor. The options are a floor field in the manifest (a schema change by its owners), the AR and IAM versions the manifest pairs per GitLab release (which criterion 5's check reads), or a document AR publishes. This issue's description records the choice and its date. 1. Decision for product, since no spec covers it: the upgrade rule that the zero-downtime policy, as the 2026-09-30 comment states it, implies. Either the AR release bundled with a GitLab minor works with the IAM release bundled with the previous minor, so AR adopts an IAM RPC one GitLab release after IAM ships it, or the documented upgrade moves IAM before AR. `## Upgrading` in `docs/dev/self-managed-install.md` states the rule. 1. Decision for the S25 owner, since S25 states no minimum IAM version: the file holding AR's IAM floor, which criteria 4 and 5 read, and the rule that sets it. At `eca9f1bab` the floor is v1.49.0, the first IAM release serving every RPC in `rpcMethods`. The vendored proto version cannot stand in, because v1.24.0 declares `LookupResources` and no IAM served it before v1.49.0. The amended `### Proto schema sourcing` names the file and the rule. 1. `test:integration:iam-rig` runs the `internal/iam` integration suite at the floor, read from criterion 3's file, and at the newest tag it tracks today, which catches an IAM release that drops an RPC AR calls. Its `changes:` list includes the floor file. The MR that lands this records a run against v1.48.0 in which `TestClient_IntegrationLookupResources` fails with `Unimplemented`. S25 acceptance criterion 14 adds `LookupResources`. 1. Decision for the AR maintainers, since no spec covers it: the check that fails before an AR release reaches an IAM below its floor, and where it reads each target's IAM version. On GitLab.com, production waits on a manual play (`docs/dev/releases.md`). IAM's deployed version there sits in a private Runway deployment project that no AR job reads. On Self-Managed and Dedicated, the target is the IAM version that criterion 1's document pairs with the AR release. This issue's description names the check and its targets. The MR that adds the check records a failing run with the floor set above a target's version. 1. Decision for the S25 owner, since `### Connection` points the probe at `grpc.health.v1.Health/Check`: whether the client also reads IAM's version from `LookupService.Health` and compares it with the floor. IAM reports that version from v1.38.0. AR's v1.24.0 stubs have no field for it, so adopting this bumps the vendored proto. S25 `## Dependencies` rules out a readiness gate, so a version below the floor yields only a log line and a metric. An empty version, which IAM documents for non-release builds, counts as unknown. With the comparison in place, a unit test against an in-process server that reports v1.48.0 asserts both signals. 1. Decision for the S25 owner, since `### Errors` routes `UNIMPLEMENTED` through its any-other-code row: a row mapping it to `iam.ErrUnimplemented`, matching `glaz.ErrUnimplemented`. With the row in place, acceptance criterion 4's table test covers it: an in-process server answering `codes.Unimplemented` yields an error that matches the sentinel through `errors.Is` and keeps the status for `errors.As`. 1. Decision for the S09 owner, since `docs/dev/observability.md` defines `transport` as any `LookupResources` call IAM refused or could not answer: whether an `Unimplemented` drain books its own `listing_failure_reason`. A new value changes that row, `drainLookupFailureReason`, and the closed set with its budget of 5 (`internal/metrics/cardinality.go:319`). Keeping `transport` leaves the IAM client series' `grpc_code` to tell the two apart. The response stays the 503 S09 specifies either way. 1. `docs/dev/alerting.md` gains an IAM client entry, in the catalog's form (series, condition, and the readings that are not a fault), for `gitlab_artifact_registry_iam_data_access_requests_total{grpc_code="Unimplemented"}` rising at all. The entry names the RPC from the `rpc` label. 1. Once product rules on criterion 1, row 9 of `docs/dev/self-managed-limitations.md` and the no-matrix bullet in `## Scope` of `docs/dev/self-managed-install.md` link the published floor. ## Specs and ADRs - [S25, Errors](https://gitlab.com/gitlab-org/ops/artifact-registry/-/blob/main/docs/specs/S25-iam-relationships-client.md#errors): the status-to-sentinel table criterion 7 amends. - [S25, Proto schema sourcing](https://gitlab.com/gitlab-org/ops/artifact-registry/-/blob/main/docs/specs/S25-iam-relationships-client.md#proto-schema-sourcing): the vendoring rule criterion 3 extends with a floor. - [S25, Connection](https://gitlab.com/gitlab-org/ops/artifact-registry/-/blob/main/docs/specs/S25-iam-relationships-client.md#connection): the health probe criterion 6 extends. - [S25, Dependencies](https://gitlab.com/gitlab-org/ops/artifact-registry/-/blob/main/docs/specs/S25-iam-relationships-client.md#dependencies): the IAM health signal never gates readiness. - [S25, Acceptance Criteria](https://gitlab.com/gitlab-org/ops/artifact-registry/-/blob/main/docs/specs/S25-iam-relationships-client.md#acceptance-criteria): criterion 4 (error mapping) and criterion 14 (integration test). - [S09, Repository listing](https://gitlab.com/gitlab-org/ops/artifact-registry/-/blob/main/docs/specs/S09-authorization.md#repository-listing): the drain, and the 503 on any evaluation failure. - [S09, Observability](https://gitlab.com/gitlab-org/ops/artifact-registry/-/blob/main/docs/specs/S09-authorization.md#observability): the listing filter's failure series. - [S09, Dependencies](https://gitlab.com/gitlab-org/ops/artifact-registry/-/blob/main/docs/specs/S09-authorization.md#dependencies): the GLAZ `BatchCheck` requirement, the one version requirement a spec states. - [ADR-021, Constraints](https://gitlab.com/gitlab-org/ops/artifact-registry/-/blob/main/docs/adr/021_authorization.md#constraints): authorization fails closed when a dependency it needs is unavailable. - [ADR-021, Grant-bounded repository-list evaluation](https://gitlab.com/gitlab-org/ops/artifact-registry/-/blob/main/docs/adr/021_authorization.md#grant-bounded-repository-list-evaluation): the `LookupResources` adoption. - [ADR-024](https://gitlab.com/gitlab-org/ops/artifact-registry/-/blob/main/docs/adr/024_infrastructure_delivery.md): AR and IAM as separately built and deployed components. - [Release manifest design](https://handbook.gitlab.com/handbook/engineering/architecture/design-documents/release_manifest/): one compatible version per component for each GitLab release. ## Dependencies - The GitLab-to-AR half asks product to choose between a per-pair matrix and the release manifest's compatibility contract, which criterion 1 also weighs, so one ruling can settle both. It has no code ordering: gitlab-org/ops/artifact-registry#1468+s - The zero-downtime upgrade procedure carries criterion 2's rule, so it follows criterion 2: gitlab-org/ops/artifact-registry#1329+s - If criterion 6 adopts runtime detection, its metric reaches alerting through the signal registry that S25 `## Dependencies` assigns to gitlab-org/ops/artifact-registry#58+s. - Under criterion 1's manifest-field option, the schema change belongs to the manifest's owners. The [schema project](https://gitlab.com/gitlab-org/release/manifests/schema) had no issue for it on 2026-10-01, so file one there before criterion 5's check depends on the field. - The GLAZ action-catalogue parity test covers the same drift on the GLAZ edge, with no ordering: gitlab-org/ops/artifact-registry#1301+s - The e2e work that the original description's authz arm belongs to has no ordering, because criterion 4 uses `test:integration:iam-rig`, which `main` carries at `eca9f1bab`: gitlab-org/ops/artifact-registry#949+s ## First step An S25 amendment MR comes first, since criteria 3, 6, and 7 are S25 rulings that wait on no product decision: the floor's file and rule at v1.49.0, the `UNIMPLEMENTED` row, the ruling on reading IAM's version, and `LookupResources` in acceptance criterion 14. A plan MR then sequences criteria 4, 5, 8, and 9, starting with the floor file, `iam.ErrUnimplemented`, and the rig at the floor. Product's rulings on criteria 1 and 2 gate only criterion 10, the install guide's upgrade rule, and the Self-Managed and Dedicated half of criterion 5. ## Original description ## Summary AR can release a call to an IAM RPC that the deployed IAM does not implement. Nothing in the build, CI, or deploy path compares AR's requirements against the IAM version it will run against. Reported from production: > The namespace itself was provisioned, the problem is that IAM replied with an > error (the version deployed does not have the new LookupResources operation) AR failed closed, which is correct behavior. The gap is that nothing caught it earlier, and nothing named it when it happened. ## Where the versions sit ``` AR's code needs >= 1.49.0 | v IAM ---- 1.44.0 ---------- 1.49.0 ---------- 1.53.1 ---- 1.54.1 ----> | | | | e2e authz arm LookupResources CI iam-rig production (!2052) implemented .gitlab-ci (Runway) | below the floor: cannot serve the RPC AR calls contract: AR's vendored IAM proto is pinned at 1.24.0 (2026-07-02). It declares LookupResources nine weeks before any server implemented it. ``` Measured 2026-09-11; AR at `origin/main` `d30ace636`. | | Pinned where | Version | | --- | --- | --- | | AR's vendored proto | `vendir.yml` | 1.24.0 | | e2e authz arm | `test/caproni/fragments/caproni.authz.yaml:32` | 1.44.0 | | CI `test:integration:iam-rig` | `.gitlab-ci.yml:1241` | v1.53.1, Renovate chasing v1.53.4 | | Production | Runway deployment project, Flux manifest | 1.54.1 | | **AR's code requires** | nowhere | **>= 1.49.0** | Four independent pins. None of them references another, and the requirement in the last row is not written down anywhere. ## Why nothing catches it 1. **The proto declares before the server implements.** `lookup.proto:33` declares `rpc LookupResources` at the pinned v1.24.0; IAM implemented it in v1.49.0. `buf:generate-check` only proves stubs match protos. 1. **AR and IAM deploy independently, several times a day.** Neither pipeline reads the other's version. On 2026-09-11 AR moved 2.38.3 to 2.42.0; IAM moved 1.49.0 to 1.54.1 over five business days, deployed minutes after each upstream tag. 1. **No test runs a version that would fail.** Unit tests use a fake, which the plan states plainly: "the fake returns the empty page, so the suite cannot catch a pre-fix build; only the re-confirmation can" (`docs/plans/2026-09-04-lookup-resources-list-visibility.md`). The two environments that run a real IAM pin versions chosen by hand. 1. **The failure has no name.** `internal/iam/errors.go:45-55` maps five gRPC codes. `Unimplemented` is not one of them, so it arrives as an opaque `iam: rpc:` error and a 503 indistinguishable from any other RPC failure. 1. **Runtime detection is not available.** IAM's `HealthResponse` carries only `string status`, and gRPC reflection is disabled in staging and production (`enable_reflection = false`). AR's health check reports `SERVING` against an IAM that cannot serve the call. ## For contrast, GLAZ does not have this problem `scripts/chart-gen.sh` generates the chart from `.runway/fairway.yaml`, so production and the e2e authz arm take the GLAZ sidecar from the same pin (`.runway/fairway.yaml:137`, v6.0.1), and `vendir.yml` states that the vendored GLAZ proto tracks that sidecar version. One pin, one contract, both environments. ## Notes - While !2052 is open, `main`'s e2e environment has no authz arm. Once it merges, the arm deploys a real GLAZ sidecar and a real `iam-data-access` at the version above. - The `iam-data-gke-grpc` deployment record shows production at v1.49.0 or later since 2026-09-04 09:23, and AR wired `LookupResources` to a live route on 2026-09-10 (`1cddbec14`). The reported instance therefore needs its environment and timestamp to reproduce. The version-comparison gap holds either way. ## Related, for context - #1301, #1297 - the same class of drift between AR and GLAZ, on action names - #949 - the e2e work the authz arm belongs to - #969 - the `LookupResources` adoption