Daily sync worker in cells for bundled catalog components

Daily sync worker in cells for bundled catalog components

Problem

Customer cells need GitLab-official CI components delivered into their own local object storage and cell-local catalog tables so that pipeline creation on the customer cell can read them without making any runtime cross-cell calls. A recurring sync worker pulls new bundles and catalog metadata from the default-organization API and stamps them into the cell-local catalog_bundled_* tables. "Official" here means catalog resources with verification_level == :gitlab_maintained — the same set served by the default-organization API (#599878).

Scope of work

  • Add a daily Sidekiq cron worker that runs on each customer cell.
  • The worker fetches the full list of available bundled versions from the default-organization API (#599878) and compares it with the cell's catalog_bundled_* rows. It fetches the bundle of every listed version the cell does not have, and removes every local version that is no longer listed. The set of components advertised by that API is constrained to verification_level == :gitlab_maintained; the worker does not re-check eligibility — it trusts the API's filter.
  • For each version the cell does not have:
    • Write the bundle artifact into the customer cell's own object storage.
    • Upsert into the cell-local catalog_bundled_resources table (keyed by server_fqdn / full_path), catalog_bundled_resource_versions (keyed by its (catalog_bundled_resource_id, semver) natural key), and catalog_bundled_resource_components. verification_level on the local record is set to :gitlab_maintained to match the source-of-truth value on the default cell.
  • Transactional / ordering pattern for partial-failure retry: write the bundle artifact to object storage first, then upsert the catalog rows referencing it. On retry, idempotent upsert on the natural-key UNIQUE indexes makes both writes safe to repeat. Avoid the inverse order (a catalog row referencing a not-yet-written artifact).
  • Authentication: the worker authenticates to the default-organization API via the public auth path (the same as any gitlab.com client), not via an internal cell-to-cell trust relationship.
  • Removal safety: remove local versions only after receiving a complete, successful list. An API error, a partial response or an empty list never removes anything. If one run would remove more than half of the cell's versions, log it and remove nothing.
  • A fresh cell, a newly eligible component, and a version deleted or yanked on the default cell all need no special handling: they are versions that are missing locally, or no longer listed.
  • The pipeline-creation read path on customer cells is the same code as on the default cell (#600464 (closed)); no customer-cell-specific read code is needed.

Cells-architecture alignment

This worker is sanctioned cross-organization compute under the cells development principles: it is a recurring cron job and it is idempotent. Writes target gitlab_main_cell_local tables, which have no sharding key and no organization scoping — the worker does not use Current.organization for these writes. The cross-organization touchpoint is the API call to the default organization, which is the only cross-org operation.

Acceptance criteria

  • A customer cell's "GitLab maintained" catalog tab shows official components after the worker has run, served from the cell-local catalog_bundled_* tables.
  • A pipeline on the customer cell using an official component creates successfully using only local object storage (no cross-cell calls at runtime).
  • New versions published on the default cell appear on the customer cell within one sync interval, and versions removed on the default cell disappear within one sync interval.
  • The worker is idempotent and safely retryable.
  • Partial-failure (for example, bundle written but catalog row not upserted) is recoverable on the next run without manual intervention.

Dependencies

Blocked by #599877 (closed) (schema), #630810 (closed) (publish-time collection, transitively via the API), #600463 (backfill), and #599878 (API endpoint).

Edited by Avielle Wolfe