Introduce a component database framework

Jason Plumrequested to merge
jp/many-databases-single-node into master

What does this MR do?

Introduce a component-database framework for the Linux package

Summary

Adds a generic framework — postgresql['component_databases'] — by which components bundled with (or shipped alongside) the Linux package can attach their own PostgreSQL database to the existing GitLab PostgreSQL cluster. Operators declare entries under postgresql['component_databases'] in gitlab.rb; gitlab-ctl reconfigure materialises the role, database, extensions, and (when PgBouncer is enabled) the pool entry, pg_shadow_lookup auth function, and pg_auth line for each entry. gitlab-ctl pgb-notify is extended so Patroni failovers propagate the new primary's host to every registered component database alongside the Rails databases.

The feature is opt-in. Default installs (no postgresql['component_databases'] declared) are completely unaffected — every code path is a no-op until at least one entry is registered.

What this MR is — and is not

This is a framework, not a component feature. The MR ships no application logic of its own; it gives future component teams a stable, reviewed, supported on-ramp so they don't each invent their own PostgreSQL provisioning, pooling, and failover wiring inside their cookbooks. Today, anything that isn't part of the Rails decomposition either ships its own ad-hoc cookbook (Mattermost, Registry) or has no first-class story at all. This MR closes that gap with one pattern.

The one rule for consumers

A consumer of this framework MUST support PostgreSQL WAL-based replication, in the same shape GitLab Geo already relies on for the Rails databases.

That is the only architectural eligibility test. The framework is single-physical-cluster by definition: every registered database lives on the same PostgreSQL instance (or Patroni cluster) that already hosts the Rails databases, and rides the same Patroni failover, the same Consul watcher, and — in Geo-enabled deployments — the same WAL stream out to secondaries. A component whose schema cannot be replicated via WAL (a non-Postgres store, an external service, a database with logical-only replication needs) is not eligible to use this framework and must continue to ship its own infrastructure.

Two concrete components that this work has been validated against:

  • OpenBao — fork-of-Vault secret store, configurable to use PostgreSQL as its backend with WAL-compatible writes. Independently verified to participate cleanly in Patroni replication.
  • GATE / GLAZ — internal API-gateway services that need their own PostgreSQL schema for routing/auth state. Independently verified WAL-compatible.

Both are used as the worked examples throughout the test plan and operator documentation. They are illustrative — the framework itself is component-agnostic.

Operator-facing surface

Minimal config:

postgresql['component_databases'] = {
  'gate' => {
    'enable'     => true,                    # required
    'user'       => 'gate',                  # required — PG role
    'password'   => 'gatesekrit',            # plaintext, raw md5 hex, or md5<hex>
    'database'   => 'gate_production',       # optional; defaults to the key
    'extensions' => ['pg_trgm'],             # optional
    'owner'      => 'gate_admin',            # optional; defaults to `user`
  }
}

When PgBouncer is enabled, the same entry is automatically routed and authenticated:

pgbouncer['enable'] = true
postgresql['pgbouncer_user_password'] = '<md5 hex>'
postgresql['md5_auth_cidr_addresses']  = %w(127.0.0.1/32)

HA topology — no extra config needed

On a node where PgBouncer and Patroni live on different hosts, operators already declare a Rails-DB pool entry pointing at the Patroni primary. The framework's auto-merged component-DB entries inherit host and port from that Rails entry, so component databases see the same primary-pointing address Rails uses, and pgb-notify failovers update them all together. No per-entry override is needed for the common HA shape. Single-node installs without a Rails pool entry fall back to the local PG instance (127.0.0.1:<port>).

Fetching secrets without putting them in gitlab.rb

Each entry accepts an optional extra_config_command pointing at an external script (Vault, AWS Secrets Manager, gcloud secrets, an internal vault wrapper). The script runs at reconfigure time; its stdout is parsed as YAML and merged into the entry before validation. JSON is a strict subset of YAML and is accepted unchanged, so a fetcher emitting either format works without translation.

'gate' => {
  'enable' => true,
  'user'   => 'gate',
  'extra_config_command' => '/etc/gitlab/fetch-gate-secret',
  # password is supplied by the script — no plaintext in gitlab.rb
}

Security posture: output is parsed via YAML.safe_load (!ruby/object:… is rejected); command stdout is never echoed to logs, even on failure, because it carries the secret payload — only stderr and exit code surface in error messages. The script runs as the user invoking gitlab-ctl reconfigure (typically root); operator is responsible for the script's mode/ownership.

Overrides and opt-outs

  • pgbouncer['databases']['<db>'] = {...} — operator-supplied pool entry wins over the framework default.
  • pgbouncer['pool_component_databases'] = false — disable the pool auto-merge wholesale (PostgreSQL still provisions roles/dbs/extensions).
  • enable: false on a per-entry basis — entry is treated as if absent: no role, no database, no pool entry, no auth function, no failover propagation.

Full operator guide in TESTING.md (to be folded into doc/settings/ before merge).

How it works under the hood

The framework is identity-only. The library (ComponentDatabaseRegistry) has no knowledge of PgBouncer, Patroni, Consul, backup, or Geo. Each consumer reads the framework's enabled entries and applies its own policy through its own attributes:

  • postgresql::managed_databases — iterates the framework and creates the PG role, database, and extensions per entry. Pure DB provisioning, replica-safe.
  • pgbouncer::enable — iterates the framework and merges default pool settings (inheriting host/port from the Rails pool entry; operator-supplied entries always win) into pgbouncer['databases'].
  • pgbouncer::user — auto-instantiates pgbouncer_user (the pg_shadow_lookup auth function + role grant) for every registered entry.
  • gitlab-ctl pgb-notify — reads the framework's enabled entry names from public_attributes, rewrites databases.ini to point every entry at the new primary on failover, keeps each entry's dbname (so cross-entry routing stays clean even when --pg-database targets one of them).

This separation means new consumers (Geo participation, backup integration, monitoring) attach later through their own attribute namespaces without changing the framework schema.

Components touched

File Change
files/gitlab-cookbooks/postgresql/libraries/component_database_registry.rb new — parse/validate/normalize logic, MD5 normalisation, extra_config_command execution + safe YAML merge.
files/gitlab-cookbooks/postgresql/resources/managed_database_objects.rb new — Chef resource: user + database + extensions per entry.
files/gitlab-cookbooks/postgresql/recipes/managed_databases.rb new — recipe iterating the framework.
files/gitlab-cookbooks/postgresql/recipes/standalone.rb + files/gitlab-cookbooks/patroni/recipes/enable.rb include the new recipe.
files/gitlab-cookbooks/pgbouncer/recipes/enable.rb pool auto-merge with HA host/port inheritance.
files/gitlab-cookbooks/pgbouncer/recipes/user.rb auto-auth iteration.
files/gitlab-cookbooks/pgbouncer/attributes/default.rb new pool_component_databases opt-out (default true).
files/gitlab-cookbooks/gitlab/libraries/postgresql.rb wire ComponentDatabaseRegistry.parse_variables into parse_variables.
files/gitlab-cookbooks/gitlab/libraries/helpers/pg_helper.rb expose component_databases in public_attributes.
files/gitlab-ctl-commands-ee/lib/pgbouncer.rb failover propagation extended; per-entry dbname.
spec/chef/support/shared_context/recipes_shared_context.rb new recipe in the default-loaded list.

Verification

Chefspec

Spec Coverage
spec/chef/cookbooks/postgresql/libraries/component_database_registry_spec.rb validation, MD5 normalisation (incl. md5<hex> strip), defaults, enabled_entries/names/users/owners, extra_config_command (YAML, JSON-as-YAML, override semantics, ENOENT, non-zero exit, stdout-not-leaked-on-failure, invalid YAML, non-mapping output, class-tagged YAML rejection, symbol-key normalisation).
spec/chef/cookbooks/postgresql/recipes/managed_databases_spec.rb recipe converges expected resources; replica-guard; explicit owner; nil-password path.
spec/chef/cookbooks/gitlab/libraries/helpers/pg_helper_spec.rb public_attributes deep-merge-safe with nested component_databases.
spec/chef/cookbooks/pgbouncer/recipes/pgbouncer_spec.rb pool merge, operator override precedence, opt-out, key fallback, HA host/port inheritance from the Rails entry, single-node fallback.
spec/chef/cookbooks/pgbouncer/recipes/pgbouncer_user_spec.rb auto-auth iteration over the framework.
spec/chef/gitlab-ctl-commands-ee/lib/pgbouncer_spec.rb component-DB failover propagation, multi-DB dbname correctness, malformed/missing entries.

Full aggregate spec suite passes — 510+ examples across the touched cookbooks, 0 failures.

Practical (Docker)

End-to-end verification against the live gitlab-ee container, scripted as docker-compose scenarios. The two worked components — OpenBao and GATE — appear together in dc-gate-openbao as the primary end-to-end test.

Scenario What it exercises
dc-gate-openbao Two enabled component DBs (GATE + OpenBao), end-to-end PgBouncer connect + failover.
dc-opt-out pool_component_databases = false.
dc-operator-override Operator-supplied pgbouncer['databases'] entry wins over framework default.
dc-baseline Non-regression — no component_databases, pgbouncer off.
dc-no-pgbouncer Framework set, pgbouncer disabled — auto-auth correctly gated.
dc-disabled-entry One enabled, one disabled — disabled is fully absent.
dc-md5-prehashed md5<hex> operator input is honoured verbatim.
dc-bad-config Reconfigure fails with Component database 'X' is missing required field 'user'.

A separate ha-tests/patroni-failover-pgbouncer/ scenario exercises the HA shape end-to-end: PgBouncer and Patroni on different hosts, Rails-DB pool entry pointing at the primary, component-DB entries inheriting host/port from it, primary fail-over via gitlab-ctl pgb-notify, component-DB traffic following the new primary.

CI

Pipelines on the branch:

  • Chefspec pipeline (2574200176) — success, merge-result against the post-restructure branch HEAD (6e3dbd8a7 as second parent).
  • Trigger:ee-package downstream (2574557131) — success, 79 min runtime.

Documentation

TESTING.md is the operator guide: minimal config, PgBouncer config, override knobs, opt-outs, extra_config_command (including security posture), HA inheritance, failover behaviour, ownership model, adoption of pre-existing databases, and explicit out-of-scope notes (Geo, backup, monitoring, multi-cluster). To be folded into doc/settings/ proper before merge — happy to take suggestions on the destination file.

The planning + outcome docs on this branch (PLAN_LOGICAL_DATABASES.md, MR_DESCRPTION.md) are reviewer context only and will be squash-removed before merge.

What this MR is not

  • Not a Rails decomposition change. Component databases live deliberately outside gitlab_rails['databases'] and the ALLOWED_DATABASES allowlist.
  • Not a Patroni multi-cluster change. Component databases share the local Patroni cluster by definition — that is the whole point of the WAL-replication rule.
  • Not a Consul watcher change. The existing postgresql service watcher already covers the union.
  • Not a Geo / backup / monitoring integration. The framework reserves nothing in its schema for those concerns; each integration attaches via its own component attributes when added.
  • Not a component implementation. No OpenBao cookbook, no GATE cookbook ships with this MR. Those teams pick the framework up on their own MRs once this lands.

Test plan for the reviewer

  1. Read TESTING.md — the operator-facing surface.
  2. Run the chefspec suites locally (or trust CI): 
    bundle exec rspec spec/chef/cookbooks/postgresql/ \
                  spec/chef/cookbooks/pgbouncer/ \
                  spec/chef/cookbooks/patroni/ \
                  spec/chef/cookbooks/gitlab/libraries/helpers/pg_helper_spec.rb \
                  spec/chef/gitlab-ctl-commands-ee/lib/pgbouncer_spec.rb
  3. Optional: spin up ha-tests/patroni-failover-pgbouncer/ and confirm a component DB follows a Patroni failover via pgb-notify.

Related issues

Checklist

See Definition of done.

For anything in this list which will not be completed, please provide a reason in the MR discussion.

Required

  • MR title and description are up to date, accurate, and descriptive.
  • MR targeting the appropriate branch.
  • Latest Merge Result pipeline is green.
  • When ready for review, MR is labeled workflowready for review per the Distribution MR workflow.

For GitLab team members

If you don't have access to this, the reviewer should trigger these jobs for you during the review process.

  • The manual Trigger:ee-package jobs have a green pipeline running against latest commit.
  • If config/software or config/patches directories are changed, make sure the build-package-on-all-os job within the Trigger:ee-package downstream pipeline succeeded.
  • If you are changing anything SSL related, then the Trigger:package:fips manual job within the Trigger:ee-package downstream pipeline must succeed.
  • If CI configuration is changed, the branch must be pushed to dev.gitlab.org to confirm regular branch builds aren't broken.

Expected (please provide an explanation if not completing)

  • Test plan indicating conditions for success has been posted and passes.
  • Documentation created/updated.
  • Tests added.
  • Integration tests added to GitLab QA.
  • Equivalent MR/issue for the GitLab Chart opened.
  • Validate potential values for new configuration settings. Formats such as integer 10, duration 10s, URI scheme://user:passwd@host:port may require quotation or other special handling when rendered in a template and written to a configuration file.
Edited by Hossein Pursultani

Merge request reports