Add the Artifact Registry credential to the CI job payload (monolith/S16 plan: 1/6)
What does this MR do and why?
A CI job that names an Artifact Registry image in image: or services: fails its pre-job pull today, because the runner has no credential for the Artifact Registry host. Users have to build a DOCKER_AUTH_CONFIG by hand.
This MR adds an Artifact Registry registry credential to the job payload, so the runner pulls those images with the job user's identity. It is Step 1 of the monolith/S16 plan, as monolith/S16 defines it, behind the ci_artifact_registry_pull_credentials beta flag (off by default, actor: the job's project).
- Extension point:
Gitlab::Ci::Build::Credentials::Factorygainsprepend_mod, and its EE override appendsRegistry::ArtifactRegistryto the providers. The provider has the shape of the existing two, so the payload serializer and existing runners need no change. - Conditions, in-memory first: Artifact Registry configured, flag on, the requesting runner manager reports a known executor other than Kubernetes; then two indexed reads: the organization has activated Artifact Registry (
has_oneon a unique index), and the job's user is a member of the project's organization (exists?on the unique(organization_id, user_id)index, memoized per request). A job skipped in memory pays no query. - Kubernetes is excluded because that executor writes payload credentials into a Secret in the build namespace, which other workloads there can read.
- Token:
Authn::TokenExchange::TokenIssuerfor the job's user and the project's organization, audiencesgitlab-artifact-registryandgitlab-iam-data-access, nogitlab.jobclaims. It expires at the job's start plus its timeout, at most 12 hours (the ADR-020 maximum). - Errors: the token is issued inside
valid?and memoized, so an error stays inside the provider. Any error after the in-memory checks is logged withlog_exceptionand counted, and the job runs without the credential. It is not sent to Sentry, which sends in the calling thread inside thepresent_buildphase timer, and whose rescue would swallow the phase timeout; Verify's pre-assign path logs only for the same reason.Ci::RegisterJobService::PhaseTimeoutErroris re-raised, so a request that runs out of time still drops the job withscheduler_failure. - Job Router: that endpoint serializes the job a second time after
present_build!.Ci::BuildRunnerPresenter#credentialsis memoized, so both serializations share one provider and one token. - Counter:
gitlab_ci_artifact_registry_pull_credentials_total{result, reason}, withresultissuedorskippedand a fixedreasonset, documented ingitlab_metrics.md.
Database
The provider adds three reads to the job request path, and a fourth when it issues the token, only for a job in a project with the flag on, on a runner that reports a known executor other than Kubernetes. A job skipped by those in-memory conditions runs none of them.
-- project.organization (primary key)
SELECT "organizations".* FROM "organizations" WHERE "organizations"."id" = $1 LIMIT 1;
-- organization.artifact_registry_activated? (unique index index_artifact_registry_namespace_mappings_on_organization_id)
SELECT "artifact_registry_namespace_mappings".* FROM "artifact_registry_namespace_mappings"
WHERE "artifact_registry_namespace_mappings"."organization_id" = $1 LIMIT 1;
-- user.member_of_organization? (unique index index_organization_users_on_organization_id_and_user_id; memoized per request)
SELECT 1 AS one FROM "organization_users"
WHERE "organization_users"."user_id" = $1 AND "organization_users"."organization_id" = $2 LIMIT 1;
-- TokenIssuer's organization_role claim, user.owns_organization? (index_organization_users_on_organization_id_and_user_id; memoized per request), only for an issued credential
SELECT 1 AS one FROM "organization_users"
WHERE "organization_users"."user_id" = $1 AND "organization_users"."organization_id" = $2
AND "organization_users"."access_level" = 50 LIMIT 1;They run in present_build!, after build.run! has written in the same request, so load balancing sends them to the primary. Each is a single-row lookup on a primary key or unique index. Query plans from Database Lab (gitlab-production-main):
organizationsby primary key: https://postgres.ai/console/gitlab/gitlab-production-main/sessions/82373/commands/189706artifact_registry_namespace_mappingsbyorganization_id: https://postgres.ai/console/gitlab/gitlab-production-main/sessions/82373/commands/189707organization_usersby(organization_id, user_id): https://postgres.ai/console/gitlab/gitlab-production-main/sessions/82373/commands/189708organization_usersowner check: https://postgres.ai/console/gitlab/gitlab-production-main/sessions/82373/commands/189713
References
- Related to #630573
- Step task: #638657 (closed)
- Spec: https://gitlab.com/gitlab-org/ops/artifact-registry/-/blob/main/docs/specs/monolith/S16-ci-job-image-pull-credentials.md
- Plan: gitlab-org/ops/artifact-registry!3676 (merged)
- Rollout issue: #638660
- ADR-020 (token issuance): https://handbook.gitlab.com/handbook/engineering/architecture/design-documents/artifact_registry/decisions/020_authentication_flow/
Screenshots or screen recordings
Not applicable: no UI change.
How to set up and validate locally
-
Run the specs:
bin/rspec ee/spec/lib/gitlab/ci/build/credentials/registry/artifact_registry_spec.rb \ ee/spec/requests/api/ci/runner/jobs_request_post_spec.rb \ ee/spec/requests/api/internal/ci/job_router_artifact_registry_credential_spec.rb \ spec/presenters/ci/build_runner_presenter_spec.rb \ spec/lib/gitlab/ci/build/credentials -
With Artifact Registry configured (
artifact_registry.api_urlingitlab.yml), an activated organization and a Cloud Connector key, enable the flag for a project:Feature.enable(:ci_artifact_registry_pull_credentials, Project.find_by_full_path('<path>')) -
Run a job on a Docker executor runner whose
image:names an Artifact Registry image, with noDOCKER_AUTH_CONFIG. The pull succeeds, andgitlab_ci_artifact_registry_pull_credentials_total{result="issued"}increases.
One spec note: the phase-timeout request spec raises the query threshold to 125. Dropping a job takes about 110 queries in that setup with or without the credential, over the default of 100; the spec says so in a comment.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.