Publish-time collection of component bundles

Publishing a :gitlab_maintained catalog component collects the component and every file it includes, as authored, into a bundle in object storage. That bundle is what a pipeline on a Cell reads instead of the source project, so the component runs there with the consuming pipeline's own variables, inputs, and rules.

What

CompileAndStoreService already exists and currently compiles (flattens) the component. Replace compiling with collecting and rename the service to CollectAndStoreService:

  • Walk every include: entry statically, including the include: entries of included local: files, and ignore rules: (the pipeline's own walk evaluates rules, and would drop rule-guarded files).
  • Accept only local: includes. Reject component:, remote:, project:, artifact: and template: includes, and spec:include in headers.
  • Reject dynamic include locations, where the location is only known at consume time (local: templates/$[[ inputs.lang ]].yml, local: $DIR/x.yml).
  • Reject wildcard local: locations (local: 'templates/*.yml').
  • Accept include:rules. An exists: rule must name its project with project:. For the consumer, that is project: $CI_PROJECT_PATH with ref: $CI_COMMIT_SHA (without ref: it checks the project's HEAD). ref: and regexp: are accepted alongside it.
  • Reject plain-path exists:, which inside a component checks the component's own repository instead of the consumer's (#630811, #386040 (closed)).
  • Check that the component's project is public.
  • Record the bundle's metadata: a format version, the component name, version, the collected sha, and the entry file path. Consume builds $[[ component.* ]] from it (#627209 (closed)). reference is not recorded, because it comes from what the consumer wrote in the include.
  • Write the component's bundle file through the existing ComponentUploader. The file holds the collected files at their repository paths and the metadata; define its format here, starting at format version 1.

Wire the service synchronously into Ci::Catalog::Resources::Versions::CreateService#publish, behind a feature flag: build the bundle in memory, write the object, then upsert the catalog_bundled_* rows inside the version-creation transaction. A rejection, or a failure writing the object or the rows, fails the publish, and a release that publishes to the catalog is not created (the same as when catalog validation fails). A rejection's error names the component, the file, the include and the rule.

The service is idempotent: retrying an already-collected version no-ops.

Acceptance criteria

  • Publishing a :gitlab_maintained version from a public project whose includes are all static local: includes succeeds and produces a bundle.
  • Publishing a version that collection rejects (for example a dynamic include location or a component: include) fails with an error naming the component, the file, the include and the rule, and writes no bundled rows or object.
  • Publishing a version from a non-public project behaves the same way.
  • Publishing a version with a plain-path include:rules:exists: behaves the same way.
  • When writing the object or the rows fails, the publish fails and no version is created.
  • With the feature flag off, publishing collects nothing and rejects nothing.
  • A bundle records its format version, name, version, sha and entry file path.
  • Retrying an already-published version's collection is a no-op.
Edited by Avielle Wolfe