AR can call an IAM RPC the deployed IAM does not serve, and no pipeline compares the two versions

Context

An Artifact Registry (AR) release can call an IAM RPC that the deployed IAM does not serve. Nothing before release compares the two versions. The RPC in the report, LookupResources, has one call site, the repository listing, which drains it for a caller without a namespace-level role (S09, Repository listing). IAM first serves it in v1.49.0. An older IAM answers Unimplemented, and the listing then answers the 503 that S09 specifies for any evaluation failure. The 2026-09-30 comment sets the constraint. Self-Managed and Dedicated take GitLab monthly, and the zero-downtime upgrade policy requires consecutive GitLab minors to work together, so calls between modules have to stay compatible along the monolith's release cycle. This issue serves the Dedicated deployment row (P0) of the GA epic Artifact Central: GA (.com, SM and Dedicated Co... (gitlab-org&22707) • Tim Rizzi. Its tracking issue rules out a behavior change reaching Dedicated without a communication path, because Dedicated cannot toggle feature flags: AR P0: Dedicated deployment (gitlab-org/gitlab#605684) • Unassigned. The GitLab-to-AR half of the same question is Publish and enforce a version compatibility mat... (#1468) • Unassigned.

What exists on main

Each line holds at origin/main eca9f1bab (2026-10-01) unless it names another source.

  • internal/iam/client.go: issues ReadRelationships (:130), LookupResources (:145), and grpc.health.v1.Health/Check (:159), the closed set in rpcMethods (internal/iam/metrics.go:115). Any ReadRelationships error answers 503 (internal/authz/decision.go:238). The listing's drain is the one LookupResources call site.
  • internal/iam/errors.go, mapStatus (:30): maps five codes to sentinels. Unimplemented takes the default arm (:53), a bare iam: rpc: wrap, as S25 ### Errors specifies for any other code. The GLAZ client maps it to ErrUnimplemented (internal/authz/glaz/errors.go:20), whose comment says the log line then tells a missing RPC from an outage.
  • internal/managementapi/list_authz.go: a refused drain logs a WARN (:467) and an ERROR (:243), answers 503 (:245), and books listing_failure_reason="transport" unless its own budget ran out (:474). docs/dev/observability.md:145 defines transport as a LookupResources call IAM refused or could not answer, so Unimplemented books the same value as a network fault.
  • internal/grpcutil/metrics.go:84: each IAM RPC records gitlab_artifact_registry_iam_data_access_requests_total{rpc, grpc_code}, so the reported failure counts as grpc_code="Unimplemented". docs/dev/alerting.md has no entry on that series.
  • cmd/artifact-registry/wire_iam.go:40: registers no readiness check on IAM, as S25 ## Dependencies requires. The one Client.Health call site is scripts/iam-rig-wait-healthy.
  • vendir.yml:9: vendors the IAM proto at v1.24.0 (2026-07-02), which declares LookupResources (third_party/iam-proto/proto/lookup/lookup.proto:33) and a HealthResponse carrying only status (:49). buf:generate-check (.gitlab-ci.yml:474) fails only when the committed stubs differ from these protos (docs/dev/vendored-iam-protos.md, ## CI gate).
  • gitlab-org/auth/iam, read 2026-10-01: v1.48.0's LookupService defines no LookupResources and embeds UnimplementedLookupServiceServer, so the call answers Unimplemented. v1.49.0 (2026-09-04) implements it. From v1.38.0 (2026-08-07), HealthResponse carries version and commit from the build, with the comment "Lets callers confirm a new deploy has actually rolled out before exercising it".
  • 1cddbec14 (2026-09-10, first in AR v2.30.0) wired the drain into the listing, so AR's floor is IAM v1.49.0. No file states it. Its plan (docs/plans/2026-09-04-lookup-resources-list-visibility.md) found that the v1.24.0 proto already carried the messages, "so no vendir bump is needed" (:60). The plan re-confirmed IAM's behavior by hand (:274).
  • .gitlab-ci.yml, test:integration:iam-rig (:1295): runs internal/iam's integration suite, LookupResources included, against auth/iam:v1.60.2 (:1303), the tag Renovate last moved on 2026-09-29 (f8bdb4977). It runs on main and on canonical MRs that touch its listed paths, never on the security mirror that cuts releases.
  • .gitlab/ci/e2e-caproni-functional.gitlab-ci.yml:24: test:e2e:caproni:functional runs the e2e suite on the "Caproni e2e daily" schedule and as a manual, allow_failure job on MRs and branches. Its IAM release comes from a vendored fragment that names the chart with no version (test/caproni/vendor/gitlab-caproni/fragments/caproni.iam-data-access.yaml:75), and nothing under e2e/ reads an IAM version. The 1.44.0 pin that the original description cites (test/caproni/fragments/caproni.authz.yaml) is not on main. The authz arm landed on 2026-10-01 without it, through test(e2e): the end-to-end suite in three arms o... (!2052 - merged) • Suleimi Ahmed • 19.5.
  • .runway/fairway.yaml:135 and scripts/chart-gen.sh: production and the e2e rig take the GLAZ sidecar from one pin, v10.1.0. vendir.yml holds two more GLAZ pins: the proto at v6.0.1 (:64), whose comment says it tracks the sidecar, and the role catalog at v2.0.0 (:36), whose comment says it matches the proto. Only comments tie the three together, and no two pins match.
  • docs/dev/releases.md, ## Deployment is triggered from the mirror's main pipeline: each release uploads manifests to staging automatically and to production on a manual play. No step reads an IAM version.
  • docs/dev/self-managed-install.md: the header records the one set walked together (GitLab 19.4.0, AR chart 2.82.1, IAM chart 1.55.3, 2026-09-22). ## Scope records that no matrix exists. ## Upgrading upgrades AR before IAM, names no required order, and reads IAM's version from gitlab.iam.update.v1.UpdateService/Health.
  • docs/dev/self-managed-limitations.md, row 9 of ## What should be lifted: asks for a published matrix and names this issue for the AR-to-IAM half.
  • S25 (docs/specs/S25-iam-relationships-client.md): ### Proto schema sourcing makes each proto bump an explicit commit and names no minimum IAM version. ### Connection points the probe at grpc.health.v1.Health/Check, "not the LookupService.Health RPC". Acceptance criterion 14's integration test names ReadRelationships and Health only.
  • S09 (docs/specs/S09-authorization.md), ## Dependencies (:38): states one version requirement, a GLAZ sidecar serving BatchCheck (auth/glaz v2.0.0) "confirmed before the consuming changes ship", and none for IAM.
  • The release manifest design records one compatible version per component for each GitLab release, in a private catalog until that release publishes. Its schema allows only version, sha, ref, and source per entry (additionalProperties: false), so no entry can state a requirement on another component. The public record holds only 19/4/0.json, which lists release-platform-canary alone. IAM's .gitlab-ci.yml enables the same release-platform component as AR's.

Change

AR records, in one file, its floor: the oldest IAM release that serves every RPC AR calls. A check fails before an AR release reaches an IAM below the floor. At eca9f1bab the floor is v1.49.0, set by LookupResources. test:integration:iam-rig runs at the floor, so an MR that adopts an RPC above it fails until the MR raises the floor in review. An Unimplemented reply returns a named error, as it does on the GLAZ client. An alert names the RPC. The listing keeps its 503, which the original description calls correct. Product decides where Self-Managed and Dedicated operators read the floor and which upgrade order they follow. The GitLab-to-AR half stays out of scope, and so does the GLAZ pin drift, which no issue tracks.

Acceptance criteria

  1. Decision for product, since no spec covers it: where Self-Managed and Dedicated operators read AR's IAM floor. The 2026-09-29 comment points to the release manifest, which pairs co-released versions and has no field for a floor. The options are a floor field in the manifest (a schema change by its owners), the AR and IAM versions the manifest pairs per GitLab release (which criterion 5's check reads), or a document AR publishes. This issue's description records the choice and its date.
  2. Decision for product, since no spec covers it: the upgrade rule that the zero-downtime policy, as the 2026-09-30 comment states it, implies. Either the AR release bundled with a GitLab minor works with the IAM release bundled with the previous minor, so AR adopts an IAM RPC one GitLab release after IAM ships it, or the documented upgrade moves IAM before AR. ## Upgrading in docs/dev/self-managed-install.md states the rule.
  3. Decision for the S25 owner, since S25 states no minimum IAM version: the file holding AR's IAM floor, which criteria 4 and 5 read, and the rule that sets it. At eca9f1bab the floor is v1.49.0, the first IAM release serving every RPC in rpcMethods. The vendored proto version cannot stand in, because v1.24.0 declares LookupResources and no IAM served it before v1.49.0. The amended ### Proto schema sourcing names the file and the rule.
  4. test:integration:iam-rig runs the internal/iam integration suite at the floor, read from criterion 3's file, and at the newest tag it tracks today, which catches an IAM release that drops an RPC AR calls. Its changes: list includes the floor file. The MR that lands this records a run against v1.48.0 in which TestClient_IntegrationLookupResources fails with Unimplemented. S25 acceptance criterion 14 adds LookupResources.
  5. Decision for the AR maintainers, since no spec covers it: the check that fails before an AR release reaches an IAM below its floor, and where it reads each target's IAM version. On GitLab.com, production waits on a manual play (docs/dev/releases.md). IAM's deployed version there sits in a private Runway deployment project that no AR job reads. On Self-Managed and Dedicated, the target is the IAM version that criterion 1's document pairs with the AR release. This issue's description names the check and its targets. The MR that adds the check records a failing run with the floor set above a target's version.
  6. Decision for the S25 owner, since ### Connection points the probe at grpc.health.v1.Health/Check: whether the client also reads IAM's version from LookupService.Health and compares it with the floor. IAM reports that version from v1.38.0. AR's v1.24.0 stubs have no field for it, so adopting this bumps the vendored proto. S25 ## Dependencies rules out a readiness gate, so a version below the floor yields only a log line and a metric. An empty version, which IAM documents for non-release builds, counts as unknown. With the comparison in place, a unit test against an in-process server that reports v1.48.0 asserts both signals.
  7. Decision for the S25 owner, since ### Errors routes UNIMPLEMENTED through its any-other-code row: a row mapping it to iam.ErrUnimplemented, matching glaz.ErrUnimplemented. With the row in place, acceptance criterion 4's table test covers it: an in-process server answering codes.Unimplemented yields an error that matches the sentinel through errors.Is and keeps the status for errors.As.
  8. Decision for the S09 owner, since docs/dev/observability.md defines transport as any LookupResources call IAM refused or could not answer: whether an Unimplemented drain books its own listing_failure_reason. A new value changes that row, drainLookupFailureReason, and the closed set with its budget of 5 (internal/metrics/cardinality.go:319). Keeping transport leaves the IAM client series' grpc_code to tell the two apart. The response stays the 503 S09 specifies either way.
  9. docs/dev/alerting.md gains an IAM client entry, in the catalog's form (series, condition, and the readings that are not a fault), for gitlab_artifact_registry_iam_data_access_requests_total{grpc_code="Unimplemented"} rising at all. The entry names the RPC from the rpc label.
  10. Once product rules on criterion 1, row 9 of docs/dev/self-managed-limitations.md and the no-matrix bullet in ## Scope of docs/dev/self-managed-install.md link the published floor.

Specs and ADRs

Dependencies

First step

An S25 amendment MR comes first, since criteria 3, 6, and 7 are S25 rulings that wait on no product decision: the floor's file and rule at v1.49.0, the UNIMPLEMENTED row, the ruling on reading IAM's version, and LookupResources in acceptance criterion 14. A plan MR then sequences criteria 4, 5, 8, and 9, starting with the floor file, iam.ErrUnimplemented, and the rig at the floor. Product's rulings on criteria 1 and 2 gate only criterion 10, the install guide's upgrade rule, and the Self-Managed and Dedicated half of criterion 5.

Original description

Summary

AR can release a call to an IAM RPC that the deployed IAM does not implement. Nothing in the build, CI, or deploy path compares AR's requirements against the IAM version it will run against. Reported from production:

The namespace itself was provisioned, the problem is that IAM replied with an error (the version deployed does not have the new LookupResources operation)

AR failed closed, which is correct behavior. The gap is that nothing caught it earlier, and nothing named it when it happened.

Where the versions sit

                          AR's code needs >= 1.49.0
                                      |
                                      v
  IAM  ---- 1.44.0 ---------- 1.49.0 ---------- 1.53.1 ---- 1.54.1 ---->
              |                  |                 |           |
        e2e authz arm     LookupResources      CI iam-rig   production
          (!2052)           implemented       .gitlab-ci    (Runway)
              |
        below the floor: cannot serve the RPC AR calls

  contract:  AR's vendored IAM proto is pinned at 1.24.0 (2026-07-02). It
             declares LookupResources nine weeks before any server
             implemented it.

Measured 2026-09-11; AR at origin/main d30ace636.

Pinned where Version
AR's vendored proto vendir.yml 1.24.0
e2e authz arm test/caproni/fragments/caproni.authz.yaml:32 1.44.0
CI test:integration:iam-rig .gitlab-ci.yml:1241 v1.53.1, Renovate chasing v1.53.4
Production Runway deployment project, Flux manifest 1.54.1
AR's code requires nowhere >= 1.49.0

Four independent pins. None of them references another, and the requirement in the last row is not written down anywhere.

Why nothing catches it

  1. The proto declares before the server implements. lookup.proto:33 declares rpc LookupResources at the pinned v1.24.0; IAM implemented it in v1.49.0. buf:generate-check only proves stubs match protos.
  2. AR and IAM deploy independently, several times a day. Neither pipeline reads the other's version. On 2026-09-11 AR moved 2.38.3 to 2.42.0; IAM moved 1.49.0 to 1.54.1 over five business days, deployed minutes after each upstream tag.
  3. No test runs a version that would fail. Unit tests use a fake, which the plan states plainly: "the fake returns the empty page, so the suite cannot catch a pre-fix build; only the re-confirmation can" (docs/plans/2026-09-04-lookup-resources-list-visibility.md). The two environments that run a real IAM pin versions chosen by hand.
  4. The failure has no name. internal/iam/errors.go:45-55 maps five gRPC codes. Unimplemented is not one of them, so it arrives as an opaque iam: rpc: error and a 503 indistinguishable from any other RPC failure.
  5. Runtime detection is not available. IAM's HealthResponse carries only string status, and gRPC reflection is disabled in staging and production (enable_reflection = false). AR's health check reports SERVING against an IAM that cannot serve the call.

For contrast, GLAZ does not have this problem

scripts/chart-gen.sh generates the chart from .runway/fairway.yaml, so production and the e2e authz arm take the GLAZ sidecar from the same pin (.runway/fairway.yaml:137, v6.0.1), and vendir.yml states that the vendored GLAZ proto tracks that sidecar version. One pin, one contract, both environments.

Notes

  • While !2052 (merged) is open, main's e2e environment has no authz arm. Once it merges, the arm deploys a real GLAZ sidecar and a real iam-data-access at the version above.
  • The iam-data-gke-grpc deployment record shows production at v1.49.0 or later since 2026-09-04 09:23, and AR wired LookupResources to a live route on 2026-09-10 (1cddbec14). The reported instance therefore needs its environment and timestamp to reproduce. The version-comparison gap holds either way.

Related, for context

  • #1301, #1297 (closed) - the same class of drift between AR and GLAZ, on action names
  • #949 (closed) - the e2e work the authz arm belongs to
  • #969 - the LookupResources adoption
Edited by Hayley Swimelar