## Problem The design of the current release and publishing process for components projects has known limitations: - During the creation of a release we extract metadata for each component server-side and this occurs synchronously during the release creation web request. We do this synchronously so that we can provide immediate success/failure feedback to users. - The limit is set to 30 components per release to avoid doing over processing on the server side. - The metadata we can collect and generate for components is limited to what GitLab knows and requires requests to be fired to Gitaly. - We are computationally limited on what we can do server side. ## Ideas We have been discussing several approaches ([1](https://gitlab.com/gitlab-org/gitlab/-/issues/428130 "Release process of Catalog Resources (`catalog-manifest.json`)"), [2](https://gitlab.com/gitlab-org/gitlab/-/issues/428142 "Release process of Catalog Resources (`Release REST API`)")) including https://gitlab.com/gitlab-org/gitlab/-/issues/438066+ and a PoC https://gitlab.com/gitlab-org/gitlab/-/merge_requests/144663+ showing how we could move the metadata processing on the client side by leveraging the existing `release-cli` that is the recommended tool to release and publish CI/CD components. ## Goal Implement and rollout a new publishing process v2.0 that allows us to: - not be limited by potential computationally intensive tasks. - add more metadata collection when we need and while we release it. E.g. in the future we could generate SBOM files for component releases. - support more types of components in the future. - have the possibility to split the release creation from the publishing aspect. ## Iterations ### Before CI/CD catalog GA 1. [x] Before CI/CD catalog goes to GA we need to address https://gitlab.com/gitlab-org/gitlab/-/issues/442066+. * This makes the `release-cli` the only way to create and publish releases to the CI catalog. All other ways (direct API and UI) are blocked. * Having this in place before GA allows us to change the logic of the `release-cli` in the future without (or greatly limiting) breaking changes. 2. [ ] [Document the engineering decision](https://gitlab.com/gitlab-org/gitlab/-/issues/442067) to use client-side processing in the architecture design doc for CI Catalog. Define and communicate a clear evolution path of the publishing process. ### After CI/CD catalog GA <details><summary>old</summary> 1. [x] ~~Expose `$CI_CATALOG_RESOURCE_PROJECT` predefined variable on the Runner. Since the `release-cli` uses `$CI_JOB_TOKEN` and this token can access a limited number of endpoints today, we need a way to know whether the project is marked as a catalog resource.~~ Canceled in https://gitlab.com/gitlab-org/gitlab/-/issues/442782 2. [ ] Add new Publish API endpoint `POST :id/catalog/publish` that takes in input the released tag + components metadata. It runs basic validations and publishes the release with metadata to the catalog. https://gitlab.com/gitlab-org/gitlab/-/issues/442783 3. [ ] Change `release-cli` to extract and validate metadata if `$CI_CATALOG_RESOURCE_PROJECT` and `$CI_JOB_TOKEN` variables are set AND `$CI_SERVER_VERSION \>= X`. For `X` we intend the version that exposes the new Publish API and the `$CI_CATALOG_RESOURCE_PROJECT` variable. Otherwise follow the previous/existing code path. https://gitlab.com/gitlab-org/gitlab/-/issues/442785 * The new `release-cli` logic would use the Release API without the `catalog_publish` parameter because we just want to create a release. After the release is successfully created, the CLI uses the Publish API. * We could be working on a dev branch of the `release-cli` project and merge MRs in there and not `master`. From there we could have experimental builds (Docker images and binaries) that we can test and dogfood before merging to `master` and cut an official major release. 4. [ ] Merge `release-cli` changes to `master` and, cut new major release with Docker image and binaries. * Issue a deprecation warning encouraging users to use the latest version of `release-cli` without breaking changes. Explain that breaking changes will occur in %18.0 when the old code-path in Release API (and internal `catalog_publish` parameter) is removed. https://gitlab.com/gitlab-org/gitlab/-/issues/442785 5. [ ] With %18.0 remove old code-path in Release API (and internal `catalog_publish` parameter). The Release API returns to only creating releases. Users using old `release-cli` will get a 400 error that the parameter `catalog_publish` doesn't exist and they are forced to use a more recent version. https://gitlab.com/gitlab-org/gitlab/-/issues/442792 and https://gitlab.com/gitlab-org/gitlab/-/issues/427861 * Only old CLI versions would be using the `catalog_publish` parameter. The `release-cli` version with new logic would not use this parameter and it should be safe to remove at this stage. </details> Work stream dependencies: ```mermaid flowchart TD t0[Ensure publish only via release-cli] t0 --> t1[Add new Publish API] t1 --> t2[Release-cli extracts metadata and publishes] t2 --> t3[Release major version of release-cli] t3 -- in 18.0 --> t4[Breaking change: remove old logic] ``` \* step 3 regarding changes to the `release-cli` could be done in parallel and in several iterations, using mocks the API and merge changes to a dev branch until completed. ## Steps - [x] Enable this feature flag for all [GitLab](https://gitlab.com/gitlab-org) and [GitLabComponents](https://gitlab.com/components/) projects. This way, we'll see if we break anything. - We can't enable the feature flag for all `gitlab.com` projects or self-hosted instances because of the scenario defined at the "_When GitLab is new_"->"_When release-cli is old_"->"_When the FF is enabled_" in https://gitlab.com/gitlab-org/gitlab/-/merge_requests/146198#2-when-gitlab-is-new. - [x] Milestone 17.5; Implement https://gitlab.com/gitlab-org/gitlab/-/issues/442783+. - [ ] Milestone 17.5; Implement https://gitlab.com/gitlab-org/gitlab/-/issues/442785+. - [ ] Milestone 17.5; After #442785, create [a version of `release-cli`](https://gitlab.com/gitlab-org/release-cli/-/releases) (Let's say `0.?.0`). - [ ] Milestone 17.6; Create a deprecation note for 18.0 that creating a release will no longer publish the CI component to the CI Catalog. We'll suggest they calculate the metadata themselves and use the new API endpoint for publishing the CI component to the CI catalog. (targets this use-case: https://gitlab.com/gitlab-org/gitlab/-/issues/442066#note_1910121919) (https://gitlab.com/gitlab-org/gitlab/-/issues/465209+) - [ ] Milestone 17.6; Create a deprecation note for 18.0 that `release-cli < 0.?.0` will no longer work for creating a release for a CI component. We'll suggest upgrading the `release-cli` version (targets this use-case: https://gitlab.com/gitlab-org/gitlab/-/issues/443782#note_1926778306) (https://gitlab.com/gitlab-org/gitlab/-/issues/465209+) - In 17.6, we have already announced that `< 0.?.0` release-cli won't work. So, we could now do these steps => - [ ] During milestone 18.0; Enable this feature flag by default but disable it for `gitlab.com`. https://gitlab.com/gitlab-org/gitlab/-/issues/465291+ - [ ] Milestone 18.0 release date; Enable this feature flag for all `gitlab.com`. https://gitlab.com/gitlab-org/gitlab/-/issues/465292+ - [ ] Milestone 18.1; Remove the feature flag. https://gitlab.com/gitlab-org/gitlab/-/issues/443782+ - [ ] Milestone 18.2; Remove `legacy_catalog_publish` from the GitLab app. https://gitlab.com/gitlab-org/gitlab/-/issues/465293+ - [ ] Milestone 18.2; Remove `legacy_catalog_publish` from `release-cli`. https://gitlab.com/gitlab-org/gitlab/-/issues/490951+