Release process of Catalog Resources (`Release REST API`)

TL;DR

This describes alternative way to publish CI components using CI/CD instead of background Sidekiq job that has limitations on what it can consume.

This proposal focuses on adding Release REST API that would allow to publish components by the release. This is contrary to #428130 (closed) which prefers to use catalog-manifest.json.

Synopsis

We are in the process of building Global CI/CD Catalog.

Currently the release process of component resource (project) is highly implicit, not controllable enough and not future proof:

  • It happens as soon as a new Release is created
  • Is controlled by the Project Settings Toogle to enable the Project as a Catalog Resource
  • It is done be the Sidekiq in background > and requires substational amount of logic on backend to scan project
    • This raised in the past suggestions that this is performance problem and one of the reasons to change directory structure: &10728
  • Can fail and user has no way to know that
  • Can be "done" by the user by creating release

Problem

The approach to tie the scanning of repository for the components have the following trade-offs:

  • Predictability: User has no way to know whether component was published or not, since this happens automatically
  • Maintainability / Performance: Code to parse repository needs to be part of GitLab codebase, increasing maintainability cost and adding potentially expensive process for large repositories
  • Control: User has no easy way to control what is published
  • Future-proof: Adding each new component type is hard, since we "should not expect" to always have all data in a repository. We could have different component types, beyond template and step. Those could be: project template, container registry, package. All of those could use the Catalog infrastructure to easily discover those.

Proposal

We should do the following changes to the process:

  1. Stop generating manifest asynchronously on a call to Release API.
  2. We should extend Release API with POST /projects/:id/releases/:tag_name/components similar to POST evidence.
  3. We should use release-cli to call components/publish as part of the CI/CD job.
  4. The explicit call to Release API controls what is published. This is done by the CI/CD job.

Outcome

What we gain?

  • Impact: Minimal changes to the current documented process: https://docs.gitlab.com/ee/ci/components/#test-the-component.
  • Control: Full control of the process, allowing to fine tune outside of GitLab Rails codebase what is published or not.
  • Easier to maintain: the scanning is external, and can be easily extended and contributed to https://gitlab.com/gitlab-org/release-cli
  • Predictability: The result from API endpoint directly answers if component was published or not. Red pipeline while publishing is strong indicator that something failed.
  • Future-proof: Wide-support, beyond simple templates: Since the API is called, it can reference other re-usable components, like container registry images, or project templates making it simple to add new types that are beyond what is possible with Sidekiq-based scanning.

What we loose?

  • Ability for user to go to Release and create a release: this is easily to avoid if we provide a good template to .gitlab-ci.yml to execute on release creation to attach components found in repository.

Change in the process

We would do the following changes:

  1. RETAIN: We would retain a project toggle as a single source of truth if THIS project does publish components.
    • If user would like to publish component, would have to tick the project toggle to indicate that components published by releases are visible in CI/CD Catalog.
    • If user would like to stop publishing this project, it would be immediate with disable of project components.
  2. CHANGE: We would change the flow described in https://docs.gitlab.com/ee/ci/components/#test-the-component.
    1. INTRODUCE: We would extend release-cli with a syntax to publish components to use Release API.
  3. STOP: We would stop generating metadata in background (expensive)

Release YAML

# Changes the https://docs.gitlab.com/ee/ci/components/#test-the-component

include:
  - component: ./path/to/my/local/component/to/test
    inputs:
      stage: build

...

# MVC
create-release:
  stage: release
  image: registry.gitlab.com/gitlab-org/release-cli:latest
  rules:
    - if: $CI_COMMIT_TAG =~ /^v\d+/
  script:
    - >
      release-cli create \
        --description "Release $CI_COMMIT_TAG of components repository $CI_PROJECT_PATH" \
        --components-publish=true # or any other filter or any other component that we would like to publish

# With Syntatic sugar (unsure if needed, once we have CI Steps!)
create-release:
  stage: release
  image: registry.gitlab.com/gitlab-org/release-cli:latest
  rules:
    - if: $CI_COMMIT_TAG =~ /^v\d+/
  script: echo "Publish component"
  release:
    tag_name: $CI_COMMIT_TAG
    description: "Release $CI_COMMIT_TAG of components repository $CI_PROJECT_PATH"
    components: * # publish all components found

Release API addition

  • POST /projects/:id/releases/:tag_name/components to accept: tag_name, component_name, component_path, component_spec (includes inputs:), component_readme.
  • GET /projects/:id/releases/:tag_name/components - get all components published by that release
  • GET /projects/:id/releases/:tag_name/components/:id - get ID component published by that release
  • DELETE /projects/:id/releases/:tag_name/components/:id - unpublish component published by that release

References

Edited by Kamil Trzciński