Backfill bundles for existing component versions

Backfill bundles for existing component versions

Problem

Publish-time collection only runs on new component-version publishes. Every existing version of every official component is currently bundle-less. Without backfill, those versions remain inaccessible via the bundle path — Phase 1 read-from-object-storage cannot benefit them, and Phase 2 customer cells cannot use them at all. The bounded set of official components — catalog resources with verification_level == :gitlab_maintained — makes a one-time backfill tractable.

Scope of work

  • Add a one-time batch job (Sidekiq, rake task, or equivalent) that iterates over every existing version of every catalog resource with verification_level == :gitlab_maintained.
  • For each version, invoke the collect-and-store service (the same unit the publish flow uses, from #630810 (closed)).
  • Skip yanked / unpublished versions.
  • Handle collection failures with the same observability as publish. A failed backfill collection means that version remains bundle-less (no cell-local rows written); it is not advertised by the API and not used by the read path. A failed version does not fatal-abort the rest of the batch.
  • Idempotent: re-running the backfill over already-collected versions is a no-op (the artifact already exists at the predictable key; the cell-local upserts no-op on the natural-key UNIQUE indexes).

Acceptance criteria

  • After backfill completes, every non-yanked version of every official component either has a retrievable bundle at the predictable key with corresponding catalog_bundled_* rows on the default cell, or has a recorded collection failure with no cell-local rows.
  • Re-running the backfill produces no new artifacts and no errors (idempotency check).
  • Collection failures during backfill are observable.

Dependencies

Blocked by #630810 (closed) (publish-time collection). Blocks #599878 (API endpoint) — the API cannot usefully ship until backfill completes, otherwise it advertises an incomplete catalog to customer cells.

Operational note

Backfill must complete before the first customer cell goes live. On the default organization, the read-from-object-storage path can ship in parallel with in-flight backfill — versions with no artifact yet fall back to the source-project path until their backfill completes.

Edited by 🤖 GitLab Bot 🤖