Read bundled component artifacts from object storage during pipeline creation

Read bundled component artifacts from object storage during pipeline creation

Problem

When a pipeline includes an official GitLab component, GitLab currently fetches the component's YAML from the source project at the version SHA via Gitaly and resolves all transitive include: directives at runtime. This is the slow path for every pipeline that uses a component. Once bundle artifacts and cell-local catalog_bundled_* rows exist, pipeline creation can short-circuit this path by resolving directly against the cell-local tables and reading the bundle from the cell's own object storage. The same read path serves every cell — the default cell and every customer cell — with no cell-type branching.

Scope of work

This issue now covers only the reader and its finders. Wiring it into pipeline creation is #628309.

  • During pipeline creation, when an include: component: directive references an official GitLab component:
    • Resolve the component by (server_fqdn, full_path) against catalog_bundled_resources on the local cell, then locate the requested version in catalog_bundled_resource_versions and derive the object key via bundle_object_key(server_fqdn, full_path, semver).
    • If a row exists, read the artifact from the local cell's object storage and use it directly. The artifact is self-contained; no further include resolution is needed.
    • If no row exists, fall back to the existing source-project fetch path. On the default cell this covers non-official components and versions with failed/in-flight backfill. On customer cells the fallback is unreachable for non-bundled components (no source project on the cell); the include resolves to an error as it would today.
  • Feature-flag the new path for gradual rollout. Default off initially.
  • Non-official components (any verification_level other than :gitlab_maintained) are unaffected; they continue to use the source-project path regardless of feature-flag state.
  • Telemetry: record per-pipeline whether the bundle path or source-project path was used, and the corresponding latency.

Acceptance criteria

  • Gitlab::Ci::Catalog::Bundle::Reader returns the stored component document for an exact server_fqdn, full_path, semver and component_name.
  • It returns nil when no matching bundled resource, version or component row exists.
  • It returns nil, rather than raising, when the row exists but the document is missing from the cell's object storage.

The acceptance criteria for using this reader during pipeline creation — the short-circuit, fallback, feature flag, reference resolution and telemetry — moved to #628309.

Dependencies

Blocked by #599877 (closed) (schema — needs the catalog_bundled_* tables). Can deploy in parallel with backfill on the default cell — fallback handles the in-flight state. The same code path serves customer cells once #599879 (sync worker) populates their cell-local tables and object storage; no customer-cell-specific read code is required.

Implementation notes

The short-circuit must sit above the project and sha guards in Gitlab::Ci::Components::InstancePath#fetch_content!. Both guards reach the source project, and sha can fall through to project.commit(reference), a Gitaly call. A bundle read placed inside fetch_component_content has already paid the cost it exists to avoid. Everything the lookup needs is available above those guards: full_path and component_name are parsed from the address with no database access.

Ci::Catalog::BundledResources::Version.latest takes no arguments, unlike the source model's latest(major, minor). Without extending it, ~latest and partial semver references fall through to the source path every time.

The feature flag must not be enabled on a Cell until #627209 (closed) lands, or masked_blob is called with a nil project and raises.

Authorization

The bundle path does not load the source project, so InstancePath#access_allowed? cannot run. No permission check is needed on this path: the epic restricts bundling to catalog resources with verification_level == :gitlab_maintained, and those are public.

Ci::Catalog::BundledResources::CompileAndStoreService refuses to bundle a resource that is not public, so this holds as an enforced invariant rather than an assumption. If bundling is ever widened beyond :gitlab_maintained, that change has to solve authorization on the read path.

Merge requests

  • !245988 (merged) — the reader and its finders
  • The short-circuit, feature flag, fallback and telemetry are tracked in #628309
Edited by Laura Montemayor