# Bundle-based delivery of official CI components ## Problem Pipeline creation that includes a GitLab-official CI component currently resolves the component's template by fetching source-project YAML from Gitaly and recursively resolving every `include:` directive at runtime. This is slow on the default organization and is impossible on customer cells, which have no copy of the source project. A pre-compiled bundle architecture replaces runtime resolution with object-storage reads, removing the runtime resolution work on the default organization and making customer-cell support feasible. ## Background GitLab-official components live in catalog resources whose `verification_level` is `:gitlab_maintained` (the enum defined on `Namespaces::VerifiedNamespace` and mirrored onto `Ci::Catalog::Resource`). Source projects for these resources remain on the legacy cell, in the default organization, medium-term. Customer organizations are a small named set, not a fleet. The first customer cell is expected to go live in Q4 2026. Three architectural facts shape the design: object storage is per-cell rather than global; `include: local:` is permitted inside component template bodies; and schema classification is per-table, not per-row, so cell-local distribution data cannot live in org-scoped catalog tables and needs its own cell-local tables. ## Definition of "official component" For the purposes of this epic, an **official component** is a catalog resource with `verification_level == :gitlab_maintained`. This is the only level eligible for bundling. Other verification levels (`:gitlab_partner_maintained`, `:verified_creator_maintained`, `:verified_creator_self_managed`, `:unverified`) continue to use the existing source-project resolution path. ## Phase 1 — Default-organization adoption Phase 1 implements bundle-based delivery on the default cell. Component versions are compiled at publish time into self-contained YAML artifacts written to the cell's object storage; the same publish-time compile also upserts a parallel record set into new cell-local `catalog_bundled_*` tables, so the default cell participates in the bundled-browse and bundled-read paths identically to customer cells. A one-time backfill produces artifacts and bundled rows for all existing versions. Pipeline creation reads the bundle from object storage when a bundled row exists (feature-flagged for gradual rollout), with fallback to runtime source-project resolution for non-official components or in-flight backfill. Phase 1 delivers independent value: faster pipeline creation for official components and production validation of the architecture under real load. Sub-issues: 1. Compile engine for self-contained catalog bundles ([#604003](https://gitlab.com/gitlab-org/gitlab/-/issues/604003)) 2. Schema + model changes for bundled catalog resources ([#599877](https://gitlab.com/gitlab-org/gitlab/-/issues/599877)) 3. Publish-time compile of self-contained component bundles ([#600454](https://gitlab.com/gitlab-org/gitlab/-/issues/600454)) 4. Backfill compile for existing component versions ([#600463](https://gitlab.com/gitlab-org/gitlab/-/issues/600463)) 5. Read bundled component artifacts from object storage during pipeline creation ([#600464](https://gitlab.com/gitlab-org/gitlab/-/issues/600464)) ## Phase 2 — Customer-cell distribution Phase 2 extends Phase 1's foundation to customer cells. The default organization exposes a publicly accessible API serving the pre-compiled bundles. A daily Sidekiq worker on each customer cell pulls bundles via the API, writes them into the customer cell's own object storage, and upserts catalog metadata into the cell-local `catalog_bundled_*` tables. Pipeline creation on customer cells reads only from local object storage — no cross-cell calls at runtime. The sync worker is the recurring-cron + idempotent cross-organization compute pattern explicitly sanctioned by the [cells development principles](https://docs.gitlab.com/development/cells/#cells-development-principles). Sub-issues: 1. Default-organization API endpoint for bundled-component artifacts ([#599878](https://gitlab.com/gitlab-org/gitlab/-/issues/599878)) 2. Daily sync worker in customer organizations ([#599879](https://gitlab.com/gitlab-org/gitlab/-/issues/599879)) 3. Tier-gating verification for bundled components ([#599880](https://gitlab.com/gitlab-org/gitlab/-/issues/599880)) ## Goals - Make GitLab-official CI components available on customer cells. - Make pipeline creation that uses official components faster on the default organization by avoiding runtime `include:` resolution. - Validate the bundle architecture in production before customer cells go live. ## Scope **In scope:** - GitLab-official components only — catalog resources with `verification_level == :gitlab_maintained`. - Publish-time compile of self-contained bundles on the default organization. - Backfill of bundles for all existing versions of official components. - Pipeline creation on the default organization reading bundles from the legacy cell's object storage with source-project fallback. - Delivery of bundles into each customer cell's local object storage. - Catalog browse rendering of bundled records from the cell-local `catalog_bundled_*` tables (same code on every cell). **Out of scope:** - GitLab Dedicated and GitLab Self-Managed deployments. This epic targets GitLab.com (the default organization and customer cells); bundle distribution to Dedicated and Self-Managed instances is left to future work. - Catalog resources with any `verification_level` other than `:gitlab_maintained` (including `:gitlab_partner_maintained`, `:verified_creator_maintained`, `:verified_creator_self_managed`, and `:unverified`) on customer cells. - Cross-organization catalog browsing. - Publishing from customer organizations to a global catalog. - Changes to non-official-component handling on the default organization (they continue to use the source-project path). ## Open questions - **Version deletion / yanking propagation.** It is not yet clear whether component versions can be deleted or yanked after publish, and if so, how that state should propagate. With the new cell-local tables, deletion on every cell (including the default cell) flows through the same mechanism: remove or tombstone the corresponding row in `catalog_bundled_resource_versions`. On customer cells, that signal needs to ride the daily sync worker. Need to confirm with the catalog team whether deletion is a supported operation today and, if so, design the propagation mechanism. - **Two-tab Catalog browse UI.** With bundled inventory in its own cell-local tables, the existing catalog browse page (org-scoped `catalog_resources`) is unaffected by bundling. A separate "GitLab maintained" tab renders from the cell-local `catalog_bundled_*` tables and uses the same code on every cell. The UI work is a follow-up issue ([#603508](https://gitlab.com/gitlab-org/gitlab/-/issues/603508)), not in scope for [#599877](https://gitlab.com/gitlab-org/gitlab/-/issues/599877); the column set on `catalog_bundled_resource_versions` (readme, readme_html) and `catalog_bundled_resource_components` (spec) is sized to feed the component display page. ## Success criteria - All existing non-yanked component versions with `verification_level == :gitlab_maintained` have retrievable bundle artifacts after backfill completes. - A pipeline on the default organization using an official component with a bundle succeeds using the object-storage path. The same pipeline using a non-official component succeeds using the existing source-project path. - A user on any cell can browse the "GitLab maintained" catalog tab, served from the cell-local `catalog_bundled_*` tables, and see official components with accurate metadata. - A pipeline in a customer organization using an official component succeeds using only local object storage — no cross-cell calls at runtime. - No tier-gated functionality is exposed to lower-license users as a result of bundling.