## OAK Component Integration: GitLab Artifact Registry **Component:** GitLab Artifact Registry (`artifact_registry`) **Team / Group:** `group::container registry` (`devops::package`) **Epic:** [Support Rollout of Artifact Registry for Self-Managed Instances](https://gitlab.com/groups/gitlab-com/gl-infra/software-delivery/operate/-/epics/10) — this fills its "Release to Omnibus / OAK - Phase 3" slot Filed against the [OAK component integration guide](https://gitlab.com/gitlab-org/omnibus-gitlab/-/blob/master/doc/development/oak_component_integration.md) and its issue template. Artifact Registry is the first component to use that template after OpenBao. ### Related work items - gitlab-org/gitlab#599006 — the self-managed deployment-scope risk record for Artifact Registry. - gitlab-org/gitlab#590299 — "Understand OAK deployment model for Artifact Registry". This issue delivers the answers to that discovery item empirically; it may be worth closing in favour of this one. - [artifact-registry#547](https://gitlab.com/gitlab-org/ops/artifact-registry/-/issues/547) — `cluster.local` must not be assumed in self-managed setups. A colocated OAK cluster is exactly where that assumption breaks. - #9873 (make `oak` a default cookbook) and #9848 (generalize `LetsEncryptHelper` for OAK's nested attributes) — both get their forcing function from a second OAK component existing. - #9997 (closed) produced the `postgresql['component_databases']` framework this integration consumes. > [!note] > Artifact Registry on OAK is currently declared out of scope for the Closed Beta epics, which defer self-managed OAK to GA. This issue is scoped as **GA-track validation ahead of demand**, not Closed Beta scope. It exists because the working integration below is now evidence about feasibility, and because the GA scope epic does not yet carry an Omnibus/OAK deployment item alongside its CNG and Dedicated ones. --- ### Overview GitLab Artifact Registry runs in a Kubernetes cluster next to a Linux package (Omnibus) GitLab installation, so it needs OAK to bridge the two: an NGINX reverse proxy on its own hostname, the bundled PostgreSQL and Redis exposed to the cluster, a generated Helm values file, and an inferred `gitlab_rails['artifact_registry']['api_url']` so Rails can reach it. Maven and npm repositories work with this integration. OCI (`/v2/`) is blocked on the GLAZ authorization sidecar. ### Setups in scope All setups must work with the same `gitlab.rb` configuration surface (see [ADR-004](https://handbook.gitlab.com/handbook/engineering/architecture/design-documents/omnibus_adjacent_kubernetes/decisions/004_multi_node_omnibus_support/)): - Colocated (component Kubernetes cluster on the same host as a single-node Omnibus) - External cluster (component in a customer-provided cluster on separate infrastructure) - Multi-node Omnibus (settings applied on every consuming node; Helm values generated on the Rails node). `postgresql['component_databases']` must additionally be declared on every node running `pgbouncer_role` or `patroni_role`, because `gitlab-ctl pgb-notify` reads each node's local `public_attributes.json`. --- ## Step 1 — `gitlab.rb` attributes - [x] Add `oak['components']['artifact_registry']` defaults to `files/gitlab-cookbooks/oak/attributes/default.rb` - [x] Document every new attribute in `files/gitlab-config-template/gitlab.rb.template` **Attributes added:** | Attribute | Type | Default | Description | |---|---|---|---| | `oak['components']['artifact_registry']['enable']` | Boolean | not set | Enable the component integration | | `...['internal_url']` | String | `nil` | URL Omnibus NGINX proxies to (LoadBalancer / NodePort / ClusterIP); required | | `...['external_url']` | String | `nil` | Public-facing URL; sets NGINX `server_name`, listen port, and TLS mode; required | | `...['ssl_certificate']` | String | `/etc/gitlab/ssl/<fqdn>.crt` | Certificate for the HTTPS listener | | `...['ssl_certificate_key']` | String | `/etc/gitlab/ssl/<fqdn>.key` | Key for the HTTPS listener | | `...['redirect_http_to_https']` | Boolean | not set (`true` with Let's Encrypt) | Add the HTTP→HTTPS redirect block | | `...['max_upload_size_gb']` | Integer | `50` | Rendered into NGINX `client_max_body_size` and Helm `server.max_body_size` (Artifact Registry's own default is 5MB) | | `...['upload_timeout']` | Integer | `3600` | `proxy_read_timeout` / `proxy_send_timeout`; Artifact Registry raises its own read deadline to 1h on uploads | | `...['delivery_mode']` | String | `DELIVERY_MODE_PROXY` | `DELIVERY_MODE_REDIRECT` returns pre-signed object storage URLs the client must be able to reach | | `...['storage']` | Hash | `nil` | `s3` or `gcs` body, rendered verbatim into the Helm values; required, as there is no local filesystem backend | | `...['component_database_key']` | String | `artifact_registry` | Key under `postgresql['component_databases']` holding the database entry | | `...['helm_values_path']` | String | `/etc/gitlab/artifact-registry-helm-values.yaml` | Generated Helm values file | **Shared OAK attributes this component depends on.** Enabling the component opts in to PostgreSQL and Redis exposure; an explicit operator value always wins. | Attribute | Type | Default | Description | |---|---|---|---| | `oak['enable']` | Boolean | `false` | Enable OAK on this node | | `oak['network_address']` | String | `nil` | Address the cluster reaches Omnibus on; PostgreSQL and Redis bind to it; required | | `oak['cluster_source_cidrs']` | String or Array | `nil` | Range the cluster's traffic arrives from, added to `pg_hba.conf`. **Usually the node range, not the pod CIDR** — see the note under Step 2. **Reconfigure raises without it** once PostgreSQL is exposed | | `oak['expose_postgresql']` | Boolean | `nil` (component opts in) | Set `false` to keep the bundled PostgreSQL private | | `oak['expose_redis']` | Boolean | `nil` (component opts in) | Set `false` to keep the bundled Redis private | | `oak['redis_port']` | Integer | `6379` | Omnibus ships `redis['port'] = 0`, so TCP exposure needs a port | | `redis['password']` | String | none | **Reconfigure raises without it** when Redis is exposed | **Inferred settings** (operator overrides always win): `gitlab_rails['artifact_registry']['api_url']`, `npm_public_registry_url`, `token_exchange_issuer_url`, `token_exchange_keys_url` (`/oauth/discovery/keys`), and `token_exchange_audiences` (`gitlab-artifact-registry`). `api_url` is the only key in the Rails `artifact_registry` stanza. `config/gitlab.yml.example` and `config/initializers/1_settings.rb` define no `enabled` key, so enablement is whether `api_url` is set, plus the `artifact_registry_ui` and `artifact_registry_role_assignment` feature flags. --- ## Step 2 — NGINX reverse proxy - [x] Create `files/gitlab-cookbooks/oak/libraries/artifact_registry.rb` with `parse_variables` - [x] Call `Oak::ArtifactRegistry.parse_variables` from `Oak.parse_variables` - [x] Create `files/gitlab-cookbooks/oak/templates/default/nginx-gitlab-artifact_registry.conf.erb` - [x] Extend the per-component variables in `files/gitlab-cookbooks/oak/recipes/enable.rb` - [x] Extend the ChefSpec tests in `spec/chef/cookbooks/oak/recipes/enable_spec.rb` (disabled, HTTP, HTTPS, Let's Encrypt) Artifact Registry needs a dedicated hostname. It serves a catch-all `location /` because the namespace slug is the first path segment of the Maven and npm routes, so it cannot be scoped under a path prefix on the GitLab hostname. Four things in the template are load-bearing and look removable: 1. `X-Forwarded-Proto $scheme` must **override**, not forward, the client value. Artifact Registry builds its OCI bearer-challenge realm from the forwarded scheme, so passing a client-supplied header through would let an unauthenticated client downgrade the advertised token realm to cleartext. 1. Request buffering is deliberately left **on**. Artifact Registry rejects a body with no `Content-Length` (`400 SIZE_INVALID`), and `proxy_request_buffering off` would forward the client's chunked encoding verbatim. The tradeoff is that NGINX spools large uploads to disk. 1. `proxy_redirect off` — in `DELIVERY_MODE_REDIRECT` the `Location` header points at pre-signed object storage that must reach the client untouched. 1. The probe port (9090: `/-/liveness`, `/-/readiness`, `/-/metrics`) is deliberately **not** proxied. Those paths bypass all middleware including authentication and must stay cluster-internal. --- ## Step 3 — Helm values file generation - [x] Create `files/gitlab-cookbooks/oak/templates/default/artifact_registry-helm-values.yaml.erb` - [x] Auto-fill connection details from Omnibus attributes - [x] Write/delete the file at `helm_values_path` on reconfigure - [x] Document which fields are auto-filled versus operator-supplied **Auto-filled fields:** | Helm field | Source | |---|---| | PostgreSQL host, Redis host | `oak['network_address']` | | PostgreSQL port | `node['postgresql']['port']` | | Redis port | `oak['redis_port']` | | Database name, database user | `postgresql['component_databases'][component_database_key]` (`database` / `user`, falling back to the key) | | `token_exchange_issuer_url`, `token_exchange_keys_url`, `token_exchange_audiences` | `gitlab['external_url']` | | `npm.public_registry_url` | the component's `external_url` | | `server.max_body_size`, per-format upload timeouts | `max_upload_size_gb`, `upload_timeout` | | `storage.delivery_mode` | `delivery_mode` | | `storage.s3` / `storage.gcs` | `storage` (rendered verbatim) | The values file carries no secrets. The database and Redis passwords are referenced as Kubernetes secrets through the LabKit `secretRef` mechanism; object storage credentials are file paths to mounted secrets. Two notes for reviewers: - Database and Redis connectivity do **not** come from Artifact Registry's application config — that field is reserved. They come from LabKit's `infrastructure.yaml` at a hardcoded path, which the chart renders into a ConfigMap. The generated values file therefore populates both `appConfig.data` and a top-level `infrastructure` key. - The generated file sets `ARTIFACT_REGISTRY_CONFIG` explicitly. The chart sets only `CONFIG_DIR`, which Artifact Registry does not read, so without it the service starts with no configuration. --- ## Step 4 — Component database **No omnibus-gitlab code changes are needed.** The `postgresql['component_databases']` framework is generic, and per [ADR-006](https://handbook.gitlab.com/handbook/engineering/architecture/design-documents/omnibus_adjacent_kubernetes/decisions/006_no_automated_database_preparation/) the operator configures the database explicitly. **Eligibility check:** the schema must support PostgreSQL WAL-based replication. ✅ Verified against the Artifact Registry source: - **No extensions.** No `CREATE EXTENSION` in any of its goose migrations or its generated `structure.sql`. UUIDv7 primary keys are generated in Go, not by the database. - **No superuser.** Owning the database is sufficient. - **No logical-replication dependency.** Its schema decision record deliberately avoids `GENERATED AS IDENTITY`, which is logical-replication hostile. - **One database, two schemas** (`public` and `partitions`), both created by its own migrations. - [x] Confirm WAL replication eligibility - [x] Document the copy-pasteable entry in the operator documentation - [ ] Verify PgBouncer pool auto-merge (single-node and HA) — HA is out of scope for this iteration - [ ] Verify Patroni failover propagation via `gitlab-ctl pgb-notify` — out of scope for this iteration - [x] Test `extra_config_command` secret fetching **Minimum operator config:** ```ruby postgresql['component_databases'] = { 'artifact_registry' => { 'enable' => true, 'user' => 'artifact_registry', 'password' => 'changeme', 'database' => 'artifact_registry_production' } } ``` No `extensions` entry is needed. To keep the password out of `gitlab.rb`, replace `password` with `'extra_config_command' => '/etc/gitlab/fetch-artifact-registry-secret'`, whose stdout is parsed as YAML and merged into the entry at parse time. One note for Patroni: `river_leader` is an unlogged table, so it is empty on a physical standby and truncated after failover. That is ephemeral leader-election state Artifact Registry rebuilds, so it is benign. --- ## Step 5 — CI job - [x] Add a `.bats` test under `oak-tests/` covering the NGINX config and `server_name`/`proxy_pass`; `client_max_body_size`, `proxy_read_timeout`, the `X-Forwarded-Proto` override, and request buffering left on; a body larger than the NGINX 1m default not being rejected; the component database role, database, and ownership; the generated Helm values file; the inferred Rails `api_url`; and the bundled Redis and PostgreSQL exposure - [x] Register the new file in the `bats` invocation in `oak-tests/test` - [ ] Verify the `OAK:smoke-test` job passes in CI — **the BATS suite has not yet been executed anywhere** --- ## Step 6 — Operator documentation - [x] Draft the installation page (staged in this repo at `doc/administration/oak/artifact_registry.md`) - [x] Cover colocated and external-cluster setups, multi-node placement, minimum `gitlab.rb` per setup, Helm values usage, component database configuration, TLS and Let's Encrypt, and verification steps - [x] Document the known gaps below - [ ] Move the page to `gitlab-org/gitlab` under `doc/administration/` and open the docs MR **Docs MR:** _not yet opened_ --- ## Empirical verification Verified against a real package rather than only in ChefSpec, using a local Docker Compose Omnibus node with a static IP on a shared Docker network plus a k3d cluster joined to that network (harness not committed). `gitlab-ctl reconfigure` succeeds, `nginx -t` accepts the generated vhost, a request through the vhost reaches a backend, pods authenticate to the exposed PostgreSQL and Redis, and the generated values file renders cleanly through the real published Artifact Registry Helm chart. Three things were found this way that ChefSpec structurally could not catch: 1. **Reconfigure was broken.** Giving Redis a TCP port makes the redis cookbook treat it as "operator wants TCP": it disables the UNIX socket and re-points Rails at host:port, deriving the password from `redis['master_password']` — which is only inferred from `redis['password']` for Sentinel, replica, or `redis_master_role` topologies, never for the single-node case OAK targets. Rails moved to TCP with no password and reconfigure died on `NOAUTH Authentication required`. 1. **The generated Helm values file was invalid YAML.** Rendering a node attribute subtree with `to_yaml` emits `!ruby/hash:ChefUtils::Mash` tags, because `ImmutableMash#to_hash` only converts the top level. 1. **A pod's source address is not what you would guess.** A pod at `10.42.2.4` reaching the exposed PostgreSQL arrived with source `172.28.0.3` — its **node** — because the CNI masquerades traffic leaving the cluster. Granting only the pod CIDR produced `FATAL: no pg_hba.conf entry for host "172.28.0.3"`. The setting was renamed to `oak['cluster_source_cidrs']` and the documentation now states that it is usually the node range, with a `kubectl run` recipe using `inet_client_addr()` to determine it. Not verified: multi-node Omnibus, and the BATS suite itself. --- ## Known gaps 1. **OCI (`/v2/`) is not usable yet.** Container authorization requires both the `iam` and `glaz` configuration blocks plus a GLAZ sidecar; a non-FIPS GLAZ container is tracked as [artifact-registry#301](https://gitlab.com/gitlab-org/ops/artifact-registry/-/issues/301). Maven and npm work without them. 1. **PgBouncer and PostgreSQL HA are out of scope for this iteration.** Artifact Registry's migration client deliberately bypasses the pooler because goose takes a session-level advisory lock, and its infrastructure config has separate direct and pooled host pairs. This iteration points both at the primary. 1. **No `sslmode` control.** The Fairway PostgreSQL infrastructure schema has no TLS field, so the connection string carries no SSL parameter and the driver falls back to `prefer`, while the Artifact Registry database specification requires `require` or `verify-full` in production. Upstream gap. 1. **The probe port is fixed at 9090**, which Omnibus already uses for Prometheus. Relevant to a colocated deployment sharing the host network. 1. **Backups do not include component databases.** `gitlab-backup` ignores them. ## Follow-up issues to file 1. Enable OCI (`/v2/`) support for the OAK deployment (blocked on artifact-registry#301). 1. Support PgBouncer and PostgreSQL HA for the Artifact Registry component database. 1. Add `sslmode` / TLS control to the Artifact Registry database connection (upstream). 1. Make the Artifact Registry probe port configurable, or document the collision. 1. Include component databases in `gitlab-backup`. 1. Add an `AR: Self-Managed (Omnibus / OAK) deployment` item to the Artifact Registry GA scope epic, alongside its existing CNG and Dedicated deployment items. --- ## Definition of done - [x] ChefSpec passes (166 examples across the OAK cookbook specs and the `gitlab.yml` stanza spec, 0 failures; Rubocop clean) - [x] Verified end-to-end on a real package (see Empirical verification) - [ ] Multi-node Omnibus verified - [ ] CI job passes - [ ] Operator documentation published in `gitlab-org/gitlab` - [ ] PREP checklist items completed (see [PREP template](https://gitlab.com/gitlab-org/architecture/readiness/-/blob/main/templates/installation_configuration/omnibus_adjacent_kubernetes.md)) --- _{:claude: Issue drafted by Claude Code from the `oak_component_integration` template, based on an implemented and empirically verified branch. Review the scope framing in the note at the top before treating this as agreed work.}