Discussion: Separate releasing from publishing catalog resources
Problem
Mainly related to &12400 (closed).
In this issue we want to discuss whether we should change the publishing process that exists today and what this change should look like.
We have had two proposals so far: (1) Release process of Catalog Resources (`Release ... (#428142 - closed) and (2) Release process of Catalog Resources (`catalog-... (#428130 - closed). Both proposals have pros and cons that we are discussing in this issue and perhaps there can be a common proposal that takes in consideration all the tradeoffs.
Previous proposal (discarded)
(Previous) Proposal
This proposal is discarded mainly because we don't want to publish different types as separate resources. We want all types of component in a project to be versioned and published together.
To separate the release creation from the publishing process, we first create a release and then we pass the release tag as a parameter to the publishing process for a given catalog resource version.To do this, we need A REST API that understands what resource version to publish, runs basic checks and publishes it to the catalog. Client-side wrappers can be provided in the form of CI step or CLI tool to facilitate this.From an analysis of the possible catalog resource types, it seems that every resource is associated with a project: components project, container images, packages, etc. are all project-level resources. Even if in the future we want to display a "service" as a possible catalog resource, it may still be good to have a GitLab project associated with the resource, so that users can be redirected to with regards to issue tracker, code, etc.The most important advantage of having a project associated with a catalog resource is to understand the visibility and user permissions on the resource.The REST API for publishing a resource could look like this:
POST catalog/publish/:project_id/:resource_type/:version
We should allow the use of the CI_JOB_TOKEN
with the Publishing API.
The endpoint must receive:
- The
project_id
to understand where the resource will be located. - The
resource_type
to understand what type of resource we are publishing and apply the correct validations and policies (e.g. only 1 group of components can be published for a given project, multiple container images with different names can be published for a given project, etc.). - The
description
. This can be optional, if not provided we take the project description. - The
tags
orcategories
, whether free form (optional) of from a predefined list (choose 1 to 3 tags). - The
metadata
(optional), which varies per resource type being published. Component projects can indicate the list of templates and steps components in it.
Example 1: publish a new version of CI component project
-
API endpoint:
POST catalog/publish/:project_id/components/1.0
. This publishes the release with tag1.0
of the current project as new version of the component. -
Parameters:
description: "..." tags: [security, gitlab, test] metadata: # metadata can be optional but must follow the schema for the given `resource_type`. templates: - dast - sast - name: secret-detection description: "..." # optional location: path/to/secret-detection.yml # optional: if we eventually want to customize the location outside the convention. steps: [ ... ]
Example 2: Publish a new package
-
API endpoint:
POST catalog/publish/:project_id/packages/my-ruby-gem/2.1.3
. This publishes the packagemy-ruby-gem
with version2.1.3
inside the given project. -
Parameters:
description: "..." tags: [ ... ] metadata: package_type: maven checksum: "..."
Example 3: Publish a container image
-
API endpoint:
POST catalog/publish/:project_id/images/dast/1.2
. This publishes the imagedast:1.2
existing inside the given project. -
Parameters:
description: "..." tags: [ ... ] metadata: { ... }
Metadata
Metadata should be considered optional. It should be possible to publish resources without metadata (e.g. list of components in a project and their inputs) since they are not critical information for the catalog. Such information is just needed to show structured information in the UI.In the case of component inputs, the user would have already:
- defined
spec:inputs
as the ultimate SSoT. This spec will raise errors to the consumer if runtime input parameters are invalid. - documented these inputs in the
README.md
. The README is the ultimate documentation for a component project and it should be self sufficient even outside the catalog view.
⚠
Immediate actions - Need to hold off from enabling the
Components
tab in the Catalog details page because it will display metadata about components to the users, locking this UX expectation. - Need to hold off from displaying the list of components in a project in the catalog index page, for the same reason above.
We can still do this eventually but for now we can focus on changing the publishing process and then introduce metadata afterwards.
How it affects the current logic
- Existing logic: Currently the user creates a release and using the
release-cli
and as part of the release creation we publish the new version to the catalog. - New logic: The
release-cli
by default will only create a release. To publish the new release to the catalog, users have to use the new API.- Users will not see new releases in the catalog by default anymore (this is a breaking change).
- The users will gain control over which versions they want to publish. Maybe there are release candidates that they don't want to show in the catalog.
- We need to provide tooling for the publishing: either we enhance the
release-cli
to have a--publish=true
option which uses the new API after release is created or we provide a CI component that does the publishing (in the form of template and/or CI step).
Points of debate
- CircleCI and GitHub Actions ingest resources on the server-side, using the yaml files as the SSoT for metadata, inputs, outputs, etc. We are instead considering asking users to compile this information and submit it via the Publishing API. Is this the right call since the user has already crafted the
spec:inputs
very carefully.- We still need a limit in the number of components in a project that can be published in a catalog and this limit should not be very high. We need to encourage users to create small groups of cohesive components and not projects containing hundreds of unrelated components.
- We could still retain the metadata collection on server-side as async process that scans the SSoT files.
-
Relying on external scan and metadata collection, while being better in offloading this work to the client, it can present maintenance challenges because we won't control anymore what logic the users are using to publish metadata. What if we want to collect new metadata or migrate to a different format? Clients may use various versions of the client-side tool (e.g.
release-cli
or step) and it will not be easy to change, requiring a breaking change all the time to force users to use the latest version of the tool.- If we collect this metadata on the server-side we have full control over the logic and version compatibility.
- For component projects, all existing components existing for a given tag should be published at once. Allowing users to select which components to publish can alter the meaning of versioning for repository-based resources such as CI components. We should stick to always publish everything that is available for a given revision since it will anyway be available for usage via the
include:component
or the step syntax.