Schema + model changes for bundled catalog resources

Schema + model changes for bundled catalog resources

Problem

Bundled GitLab-maintained components are cell-local distribution data, not organization-owned catalog data. The existing catalog tables (catalog_resources, catalog_resource_components, catalog_resource_versions) are classified gitlab_main_org and sharded on project_id.

Schema classification is per-table, not per-row: an org-scoped table cannot hold cell-local rows. Bundled inventory needs its own cell-local tables.

This separates two data shapes:

  • Org-scoped catalog state (gitlab_main_org, unchanged): components a customer org publishes itself, and the source-of-truth project-backed rows on the default organization.
  • Cell-local bundled inventory (gitlab_main_cell_local, new): the per-cell replica of GitLab-maintained components delivered by bundling.

Scope of work

1. New cell-local tables (gitlab_main_cell_local, no sharding key)

catalog_bundled_resources

  • id: bigint, primary key — local to the cell; never used as a cross-cell reference
  • server_fqdn: text, NOT NULL — origin server FQDN of the component's source instance
  • full_path: text, NOT NULL — source project full path on the origin instance
  • name: text, NOT NULL
  • description: text
  • verification_level: smallint — gitlab_maintained for now; partner-maintained later
  • latest_released_at: timestamptz
  • search_vector: tsvector, generated (mirrors catalog_resources) — powers browse search
  • created_at, updated_at: timestamptz
  • UNIQUE (server_fqdn, full_path)

catalog_bundled_resource_versions

  • id: bigint, primary key
  • catalog_bundled_resource_id: bigint, NOT NULL — FK to catalog_bundled_resources, ON DELETE CASCADE
  • semver_major, semver_minor, semver_patch: integer
  • semver_prerelease: text
  • semver_prefixed: boolean, default false
  • released_at: timestamptz
  • readme, readme_html: text — denormalized for the component display page
  • created_at, updated_at: timestamptz
  • UNIQUE (catalog_bundled_resource_id, semver_major, semver_minor, semver_patch, semver_prerelease)

The compiled bundle's object-storage location is not stored on the version row: it is derived from (server_fqdn, full_path, semver) by a shared helper (the storage-key contract). Those columns already determine it, so persisting it would only add drift risk.

catalog_bundled_resource_components

  • id: bigint, primary key
  • catalog_bundled_resource_id: bigint, NOT NULL — FK to catalog_bundled_resources
  • catalog_bundled_version_id: bigint, NOT NULL — FK to catalog_bundled_resource_versions, ON DELETE CASCADE
  • name: text, NOT NULL
  • spec: jsonb, default {}, NOT NULL — needed for the component display page
  • created_at: timestamptz
  • UNIQUE (catalog_bundled_version_id, name)

Physical column order in the migration follows the type-size-descending ordering convention.

2. Models and db/docs

  • New models for the three cell-local tables, associated only among themselves; no organization or sharding key.
  • db/docs entries set gitlab_main_cell_local with no sharding_key (mirroring zoekt_nodes and shards).

Cells correctness (call out explicitly in the migration)

  • gitlab_main_cell_local, no sharding key — non-customer (GitLab-maintained) distribution data, distinct per cell.
  • No foreign key from any gitlab_main_org table into these tables. Org-scoped state that references a bundled component uses a loose reference by (server_fqdn, full_path), never a PK or FK.
  • Primary keys are cell-local; the same logical component has a different PK on each cell. Cross-cell identity is (server_fqdn, full_path) for a resource and (server_fqdn, full_path, semver) for a version. Foreign keys exist only within the cell-local family.
  • (server_fqdn, full_path) matches the key Gitlab::Ci::Components::InstancePath already resolves on.
  • The UNIQUE natural keys make the daily sync worker (#599879) an idempotent upsert; they are required, not optional.

Population (not implemented in this issue)

All cells operate identically. On the default cell, publish-time compile (#600454 (closed)) writes the catalog_bundled_* rows directly; on customer cells, the daily sync worker (#599879) writes them. The default cell therefore also has the "gitlab maintained" browse experience and uses the same read path (#600464 (closed)) as customer cells — no cell-type branching anywhere. This issue defines only the schema.

Out of scope

  • The two-tab ("your org" / "gitlab maintained") Catalog UI — separate issue.
  • Per-org usage analytics for bundled components — deferred. Org-scoped self-published components keep usage as today.
  • Cross-FQDN / self-managed include-resolution semantics — future work.

Acceptance criteria

  • The three catalog_bundled_* tables exist, classified gitlab_main_cell_local with no sharding key, and pass schema/sharding specs without a sharding-key exemption.
  • No foreign key references these tables from any gitlab_main_org table.
  • Each table enforces its natural-key UNIQUE index; upserting an existing logical row updates rather than duplicating.
  • A bundled resource/version/component row set can be created and queried with no project or organization association.
  • Migrations run cleanly on default-cell and customer-cell schemas.

Dependencies

None upstream. Blocks #600454 (closed) (compile populates default-cell rows), #600463 (backfill), #600464 (closed) (read path), and #599879 (sync worker populates customer-cell rows).

Edited by 🤖 GitLab Bot 🤖