OAK Component Integration: GitLab Artifact Registry

OAK Component Integration: GitLab Artifact Registry

Component: GitLab Artifact Registry (artifact_registry)

Team / Group: group::container registry (devops::package)

Epic: Support Rollout of Artifact Registry for Self-Managed Instances — this fills its "Release to Omnibus / OAK - Phase 3" slot

Filed against the OAK component integration guide and its issue template. Artifact Registry is the first component to use that template after OpenBao.

Related work items

  • gitlab-org/gitlab#599006 — the self-managed deployment-scope risk record for Artifact Registry.
  • gitlab#590299 (closed) — "Understand OAK deployment model for Artifact Registry". This issue delivers the answers to that discovery item empirically; it may be worth closing in favour of this one.
  • artifact-registry#547 — cluster.local must not be assumed in self-managed setups. A colocated OAK cluster is exactly where that assumption breaks.
  • #9873 (make oak a default cookbook) and #9848 (generalize LetsEncryptHelper for OAK's nested attributes) — both get their forcing function from a second OAK component existing.
  • #9997 (closed) (closed) produced the postgresql['component_databases'] framework this integration consumes.

Note

Artifact Registry on OAK is currently declared out of scope for the Closed Beta epics, which defer self-managed OAK to GA. This issue is scoped as GA-track validation ahead of demand, not Closed Beta scope. It exists because the working integration below is now evidence about feasibility, and because the GA scope epic does not yet carry an Omnibus/OAK deployment item alongside its CNG and Dedicated ones.

Overview

GitLab Artifact Registry runs in a Kubernetes cluster next to a Linux package (Omnibus) GitLab installation, so it needs OAK to bridge the two: an NGINX reverse proxy on its own hostname, the bundled PostgreSQL and Redis exposed to the cluster, a generated Helm values file, and an inferred gitlab_rails['artifact_registry']['api_url'] so Rails can reach it.

Maven and npm repositories work with this integration. OCI (/v2/) is blocked on the GLAZ authorization sidecar.

Setups in scope

All setups must work with the same gitlab.rb configuration surface (see ADR-004):

  • Colocated (component Kubernetes cluster on the same host as a single-node Omnibus)
  • External cluster (component in a customer-provided cluster on separate infrastructure)
  • Multi-node Omnibus (settings applied on every consuming node; Helm values generated on the Rails node). postgresql['component_databases'] must additionally be declared on every node running pgbouncer_role or patroni_role, because gitlab-ctl pgb-notify reads each node's local public_attributes.json.

Step 1 — gitlab.rb attributes

  • Add oak['components']['artifact_registry'] defaults to files/gitlab-cookbooks/oak/attributes/default.rb
  • Document every new attribute in files/gitlab-config-template/gitlab.rb.template

Attributes added:

Attribute Type Default Description
oak['components']['artifact_registry']['enable'] Boolean not set Enable the component integration
...['internal_url'] String nil URL Omnibus NGINX proxies to (LoadBalancer / NodePort / ClusterIP); required
...['external_url'] String nil Public-facing URL; sets NGINX server_name, listen port, and TLS mode; required
...['ssl_certificate'] String /etc/gitlab/ssl/<fqdn>.crt Certificate for the HTTPS listener
...['ssl_certificate_key'] String /etc/gitlab/ssl/<fqdn>.key Key for the HTTPS listener
...['redirect_http_to_https'] Boolean not set (true with Let's Encrypt) Add the HTTP→HTTPS redirect block
...['max_upload_size_gb'] Integer 50 Rendered into NGINX client_max_body_size and Helm server.max_body_size (Artifact Registry's own default is 5MB)
...['upload_timeout'] Integer 3600 proxy_read_timeout / proxy_send_timeout; Artifact Registry raises its own read deadline to 1h on uploads
...['delivery_mode'] String DELIVERY_MODE_PROXY DELIVERY_MODE_REDIRECT returns pre-signed object storage URLs the client must be able to reach
...['storage'] Hash nil s3 or gcs body, rendered verbatim into the Helm values; required, as there is no local filesystem backend
...['component_database_key'] String artifact_registry Key under postgresql['component_databases'] holding the database entry
...['helm_values_path'] String /etc/gitlab/artifact-registry-helm-values.yaml Generated Helm values file

Shared OAK attributes this component depends on. Enabling the component opts in to PostgreSQL and Redis exposure; an explicit operator value always wins.

Attribute Type Default Description
oak['enable'] Boolean false Enable OAK on this node
oak['network_address'] String nil Address the cluster reaches Omnibus on; PostgreSQL and Redis bind to it; required
oak['cluster_source_cidrs'] String or Array nil Range the cluster's traffic arrives from, added to pg_hba.conf. Usually the node range, not the pod CIDR — see the note under Step 2. Reconfigure raises without it once PostgreSQL is exposed
oak['expose_postgresql'] Boolean nil (component opts in) Set false to keep the bundled PostgreSQL private
oak['expose_redis'] Boolean nil (component opts in) Set false to keep the bundled Redis private
oak['redis_port'] Integer 6379 Omnibus ships redis['port'] = 0, so TCP exposure needs a port
redis['password'] String none Reconfigure raises without it when Redis is exposed

Inferred settings (operator overrides always win): gitlab_rails['artifact_registry']['api_url'], npm_public_registry_url, token_exchange_issuer_url, token_exchange_keys_url (/oauth/discovery/keys), and token_exchange_audiences (gitlab-artifact-registry).

api_url is the only key in the Rails artifact_registry stanza. config/gitlab.yml.example and config/initializers/1_settings.rb define no enabled key, so enablement is whether api_url is set, plus the artifact_registry_ui and artifact_registry_role_assignment feature flags.

Step 2 — NGINX reverse proxy

  • Create files/gitlab-cookbooks/oak/libraries/artifact_registry.rb with parse_variables
  • Call Oak::ArtifactRegistry.parse_variables from Oak.parse_variables
  • Create files/gitlab-cookbooks/oak/templates/default/nginx-gitlab-artifact_registry.conf.erb
  • Extend the per-component variables in files/gitlab-cookbooks/oak/recipes/enable.rb
  • Extend the ChefSpec tests in spec/chef/cookbooks/oak/recipes/enable_spec.rb (disabled, HTTP, HTTPS, Let's Encrypt)

Artifact Registry needs a dedicated hostname. It serves a catch-all location / because the namespace slug is the first path segment of the Maven and npm routes, so it cannot be scoped under a path prefix on the GitLab hostname.

Four things in the template are load-bearing and look removable:

  1. X-Forwarded-Proto $scheme must override, not forward, the client value. Artifact Registry builds its OCI bearer-challenge realm from the forwarded scheme, so passing a client-supplied header through would let an unauthenticated client downgrade the advertised token realm to cleartext.
  2. Request buffering is deliberately left on. Artifact Registry rejects a body with no Content-Length (400 SIZE_INVALID), and proxy_request_buffering off would forward the client's chunked encoding verbatim. The tradeoff is that NGINX spools large uploads to disk.
  3. proxy_redirect off — in DELIVERY_MODE_REDIRECT the Location header points at pre-signed object storage that must reach the client untouched.
  4. The probe port (9090: /-/liveness, /-/readiness, /-/metrics) is deliberately not proxied. Those paths bypass all middleware including authentication and must stay cluster-internal.

Step 3 — Helm values file generation

  • Create files/gitlab-cookbooks/oak/templates/default/artifact_registry-helm-values.yaml.erb
  • Auto-fill connection details from Omnibus attributes
  • Write/delete the file at helm_values_path on reconfigure
  • Document which fields are auto-filled versus operator-supplied

Auto-filled fields:

Helm field Source
PostgreSQL host, Redis host oak['network_address']
PostgreSQL port node['postgresql']['port']
Redis port oak['redis_port']
Database name, database user postgresql['component_databases'][component_database_key] (database / user, falling back to the key)
token_exchange_issuer_url, token_exchange_keys_url, token_exchange_audiences gitlab['external_url']
npm.public_registry_url the component's external_url
server.max_body_size, per-format upload timeouts max_upload_size_gb, upload_timeout
storage.delivery_mode delivery_mode
storage.s3 / storage.gcs storage (rendered verbatim)

The values file carries no secrets. The database and Redis passwords are referenced as Kubernetes secrets through the LabKit secretRef mechanism; object storage credentials are file paths to mounted secrets.

Two notes for reviewers:

  • Database and Redis connectivity do not come from Artifact Registry's application config — that field is reserved. They come from LabKit's infrastructure.yaml at a hardcoded path, which the chart renders into a ConfigMap. The generated values file therefore populates both appConfig.data and a top-level infrastructure key.
  • The generated file sets ARTIFACT_REGISTRY_CONFIG explicitly. The chart sets only CONFIG_DIR, which Artifact Registry does not read, so without it the service starts with no configuration.

Step 4 — Component database

No omnibus-gitlab code changes are needed. The postgresql['component_databases'] framework is generic, and per ADR-006 the operator configures the database explicitly.

Eligibility check: the schema must support PostgreSQL WAL-based replication. ✅ Verified against the Artifact Registry source:

  • No extensions. No CREATE EXTENSION in any of its goose migrations or its generated structure.sql. UUIDv7 primary keys are generated in Go, not by the database.

  • No superuser. Owning the database is sufficient.

  • No logical-replication dependency. Its schema decision record deliberately avoids GENERATED AS IDENTITY, which is logical-replication hostile.

  • One database, two schemas (public and partitions), both created by its own migrations.

  • Confirm WAL replication eligibility

  • Document the copy-pasteable entry in the operator documentation

  • Verify PgBouncer pool auto-merge (single-node and HA) — HA is out of scope for this iteration

  • Verify Patroni failover propagation via gitlab-ctl pgb-notify — out of scope for this iteration

  • Test extra_config_command secret fetching

Minimum operator config:

postgresql['component_databases'] = {
  'artifact_registry' => {
    'enable'   => true,
    'user'     => 'artifact_registry',
    'password' => 'changeme',
    'database' => 'artifact_registry_production'
  }
}

No extensions entry is needed. To keep the password out of gitlab.rb, replace password with 'extra_config_command' => '/etc/gitlab/fetch-artifact-registry-secret', whose stdout is parsed as YAML and merged into the entry at parse time.

One note for Patroni: river_leader is an unlogged table, so it is empty on a physical standby and truncated after failover. That is ephemeral leader-election state Artifact Registry rebuilds, so it is benign.

Step 5 — CI job

  • Add a .bats test under oak-tests/ covering the NGINX config and server_name/proxy_pass; client_max_body_size, proxy_read_timeout, the X-Forwarded-Proto override, and request buffering left on; a body larger than the NGINX 1m default not being rejected; the component database role, database, and ownership; the generated Helm values file; the inferred Rails api_url; and the bundled Redis and PostgreSQL exposure
  • Register the new file in the bats invocation in oak-tests/test
  • Verify the OAK:smoke-test job passes in CI — the BATS suite has not yet been executed anywhere

Step 6 — Operator documentation

  • Draft the installation page (staged in this repo at doc/administration/oak/artifact_registry.md)
  • Cover colocated and external-cluster setups, multi-node placement, minimum gitlab.rb per setup, Helm values usage, component database configuration, TLS and Let's Encrypt, and verification steps
  • Document the known gaps below
  • Move the page to gitlab-org/gitlab under doc/administration/ and open the docs MR

Docs MR: not yet opened

Empirical verification

Verified against a real package rather than only in ChefSpec, using a local Docker Compose Omnibus node with a static IP on a shared Docker network plus a k3d cluster joined to that network (harness not committed). gitlab-ctl reconfigure succeeds, nginx -t accepts the generated vhost, a request through the vhost reaches a backend, pods authenticate to the exposed PostgreSQL and Redis, and the generated values file renders cleanly through the real published Artifact Registry Helm chart.

Three things were found this way that ChefSpec structurally could not catch:

  1. Reconfigure was broken. Giving Redis a TCP port makes the redis cookbook treat it as "operator wants TCP": it disables the UNIX socket and re-points Rails at host:port, deriving the password from redis['master_password'] — which is only inferred from redis['password'] for Sentinel, replica, or redis_master_role topologies, never for the single-node case OAK targets. Rails moved to TCP with no password and reconfigure died on NOAUTH Authentication required.
  2. The generated Helm values file was invalid YAML. Rendering a node attribute subtree with to_yaml emits !ruby/hash:ChefUtils::Mash tags, because ImmutableMash#to_hash only converts the top level.
  3. A pod's source address is not what you would guess. A pod at 10.42.2.4 reaching the exposed PostgreSQL arrived with source 172.28.0.3 — its node — because the CNI masquerades traffic leaving the cluster. Granting only the pod CIDR produced FATAL: no pg_hba.conf entry for host "172.28.0.3". The setting was renamed to oak['cluster_source_cidrs'] and the documentation now states that it is usually the node range, with a kubectl run recipe using inet_client_addr() to determine it.

Not verified: multi-node Omnibus, and the BATS suite itself.

Known gaps

  1. OCI (/v2/) is not usable yet. Container authorization requires both the iam and glaz configuration blocks plus a GLAZ sidecar; a non-FIPS GLAZ container is tracked as artifact-registry#301. Maven and npm work without them.
  2. PgBouncer and PostgreSQL HA are out of scope for this iteration. Artifact Registry's migration client deliberately bypasses the pooler because goose takes a session-level advisory lock, and its infrastructure config has separate direct and pooled host pairs. This iteration points both at the primary.
  3. No sslmode control. The Fairway PostgreSQL infrastructure schema has no TLS field, so the connection string carries no SSL parameter and the driver falls back to prefer, while the Artifact Registry database specification requires require or verify-full in production. Upstream gap.
  4. The probe port is fixed at 9090, which Omnibus already uses for Prometheus. Relevant to a colocated deployment sharing the host network.
  5. Backups do not include component databases. gitlab-backup ignores them.

Follow-up issues to file

  1. Enable OCI (/v2/) support for the OAK deployment (blocked on artifact-registry#301).
  2. Support PgBouncer and PostgreSQL HA for the Artifact Registry component database.
  3. Add sslmode / TLS control to the Artifact Registry database connection (upstream).
  4. Make the Artifact Registry probe port configurable, or document the collision.
  5. Include component databases in gitlab-backup.
  6. Add an AR: Self-Managed (Omnibus / OAK) deployment item to the Artifact Registry GA scope epic, alongside its existing CNG and Dedicated deployment items.

Definition of done

  • ChefSpec passes (166 examples across the OAK cookbook specs and the gitlab.yml stanza spec, 0 failures; Rubocop clean)
  • Verified end-to-end on a real package (see Empirical verification)
  • Multi-node Omnibus verified
  • CI job passes
  • Operator documentation published in gitlab-org/gitlab
  • PREP checklist items completed (see PREP template)

Issue drafted by Claude Code from the oak_component_integration template, based on an implemented and empirically verified branch. Review the scope framing in the note at the top before treating this as agreed work.

Edited by Jason Plum