Skip to content

Release process of Catalog Resources (`catalog-manifest.json`)

TL;DR

This describes alternative way to publish CI components using CI/CD instead of background Sidekiq jobs that has inherit limitations on what it can consume.

This proposal focuses on adding catalog-manifest.json that would describe all components to publish. This is contrary to #428142 (closed) which prefers to use REST API.

Synopsis

We are in the process of building Global CI/CD Catalog.

Currently the release process of component resource (project) is highly implicit, not controllable enough and not future proof:

  • It happens as soon as a new Release is created
  • Is controlled by the Project Settings Toogle to enable the Project as a Catalog Resource
  • It is done be the Sidekiq in background > and requires substational amount of logic on backend to scan project
    • This raised in the past suggestions that this is performance problem and one of the reasons to change directory structure: &10728
  • Can fail and user has no way to know that
  • Can be "done" by the user by creating release

Problem

The approach to tie the scanning of repository for the components have the following trade-offs:

  • Predictability: User has no way to know whether component was published or not, since this happens automatically
  • Maintainability / Performance: Code to parse repository needs to be part of GitLab codebase, increasing maintainability cost and adding potentially expensive process for large repositories
  • Control: User has no easy way to control what is published
  • Future-proof: Adding each new component type is hard, since we "should not expect" to always have all data in a repository. We could have different component types, beyond template and step. Those could be: project template, container registry, package. All of those could use the Catalog infrastructure to easily discover those.

Proposal

We should do the following changes to the process:

  1. Stop generating manifest asynchronously on a call to Release API
  2. We should use release-cli to generate a catalog-manifest.json that would be uploaded as part of CI/CD job
  3. Created release, that would contain catalog-manifest.json would be parsed in-line and pulled into database as part of release creation - raising 400/401 or any other error inline when it cannot be parsed.
  4. The catalog-manifest.json would be a single entity published for federated access.
  5. The catalog-manifest.json would contain all information parsed: including spec: of each component, path to the component, README, and all dependent files required.
  6. (later) We might extend the process to publish each component as a .zip archive that would be pulled by federated GitLab instances.
  7. The catalog-manifest.json controls what is published in a catalog.

Outcome

What we gain?

  • Impact: Minimal changes to the current documented process: https://docs.gitlab.com/ee/ci/components/#test-the-component.
  • Control: Full control of the process, allowing to fine tune outside of GitLab Rails codebase what is published or not.
  • Easier to maintain: the scanning is external, and can be easily extended and contributed to https://gitlab.com/gitlab-org/release-cli
  • Contract: The catalog-manifest.json that would be generated from protobuf would represent an interface between many GitLabs, be it: who is generating it (CI job), who is consuming it (GitLab.com), who is referencing it (on-premise GitLab).
  • Predictability: The direct feedback once catalog-manifest.json is uploaded would allow to explicitly see the status of the process.
  • Future-proof: Wide-support, beyond simple templates: Since the catalog-manifest.json is generated externally, we would have opportunity to extend the system with support for other re-usable components, outside of the pure CI scripts.
  • (Optionally) Optimisation: Since what is being to be published would be generated outside we could precompile what is essential for a given component and publish it as an asset to Release to allow other systems to pull the manifest.

What we loose?

  • Ability for user to go to Release and create a release: this is easily to avoid if we provide a good template to .gitlab-ci.yml to allow to generate catalog-manifest.json once a tag is created - without a release.

Change in the process

We would do the following changes:

  1. RETAIN: We would retain a project toggle as a single source of truth if THIS project can publish components.

    • If user would like to publish component, would have to tick the project toggle to indicate that this is OK.
    • If user would like to stop publishing this project, it would be immediate with disable of project components.

  2. CHANGE: We would change the flow described in https://docs.gitlab.com/ee/ci/components/#test-the-component.

    1. INTRODUCE: We would introduce a protobuf that would describe catalog-manifest.json
    2. INTRODUCE: We would extend release-cli to provide release-cli catalog-manifest => with possibility to indicate what components should be published
    3. INTRODUCE: Once the catalog-manifest.json is attached to release we would parse it at time of upload and return information immediately.

  3. STOP: We would stop generating metadata in background (expensive)

    # Changes the https://docs.gitlab.com/ee/ci/components/#test-the-component

include:
  - component: ./path/to/my/local/component/to/test
    inputs:
      stage: build

...

# If we are tagging a release with a specific convention ("v" + number) and all
# previous checks succeeded, we proceed with creating a release automatically.
create-release:
  stage: release
  image: registry.gitlab.com/gitlab-org/release-cli:latest
  rules:
    - if: $CI_COMMIT_TAG =~ /^v\d+/
  script:
    - release-cli catalog-manifest # generate manifest for all components
    - release-cli catalog-manifest --exclude my-component-name # generate for all except
  release:
    tag_name: $CI_COMMIT_TAG
    description: "Release $CI_COMMIT_TAG of components repository $CI_PROJECT_PATH"
    assets:
      - catalog-manifest.json

Challenges with this approach

Currently release API only accepts assets links, not direct file uploads. We would have to figure out how to handle this. Probably the best would be either of those ways:

  1. Harder: Support attaching uploads to releases, and not only assets.
  2. Easier: Introduce the POST /projects/:id/releases/:tag_name/catalog_manifest similar to POST Evidence

In both cases this would require extending release-cli and YAML syntax.

References

Edited by Kamil Trzciński