Skip to content

Discussion: Separate releasing from publishing catalog resources

Problem

Mainly related to &12400 (closed).

In this issue we want to discuss whether we should change the publishing process that exists today and what this change should look like.

We have had two proposals so far: (1) Release process of Catalog Resources (`Release ... (#428142 - closed) and (2) Release process of Catalog Resources (`catalog-... (#428130 - closed). Both proposals have pros and cons that we are discussing in this issue and perhaps there can be a common proposal that takes in consideration all the tradeoffs.

Previous proposal (discarded)

(Previous) Proposal

This proposal is discarded mainly because we don't want to publish different types as separate resources. We want all types of component in a project to be versioned and published together.

To separate the release creation from the publishing process, we first create a release and then we pass the release tag as a parameter to the publishing process for a given catalog resource version.To do this, we need A REST API that understands what resource version to publish, runs basic checks and publishes it to the catalog. Client-side wrappers can be provided in the form of CI step or CLI tool to facilitate this.From an analysis of the possible catalog resource types, it seems that every resource is associated with a project: components project, container images, packages, etc. are all project-level resources. Even if in the future we want to display a "service" as a possible catalog resource, it may still be good to have a GitLab project associated with the resource, so that users can be redirected to with regards to issue tracker, code, etc.The most important advantage of having a project associated with a catalog resource is to understand the visibility and user permissions on the resource.The REST API for publishing a resource could look like this:

POST catalog/publish/:project_id/:resource_type/:version

We should allow the use of the CI_JOB_TOKEN with the Publishing API.

The endpoint must receive:

  • The project_id to understand where the resource will be located.
  • The resource_type to understand what type of resource we are publishing and apply the correct validations and policies (e.g. only 1 group of components can be published for a given project, multiple container images with different names can be published for a given project, etc.).
  • The description. This can be optional, if not provided we take the project description.
  • The tags or categories, whether free form (optional) of from a predefined list (choose 1 to 3 tags).
  • The metadata (optional), which varies per resource type being published. Component projects can indicate the list of templates and steps components in it.

Example 1: publish a new version of CI component project

  • API endpoint: POST catalog/publish/:project_id/components/1.0. This publishes the release with tag 1.0 of the current project as new version of the component.

  • Parameters:

    description: "..."
tags: [security, gitlab, test]
metadata: # metadata can be optional but must follow the schema for the given `resource_type`.
  templates:
    - dast
    - sast
    - name: secret-detection
      description: "..." # optional
      location: path/to/secret-detection.yml # optional: if we eventually want to customize the location outside the convention.
  steps: [ ... ]

Example 2: Publish a new package

  • API endpoint: POST catalog/publish/:project_id/packages/my-ruby-gem/2.1.3. This publishes the package my-ruby-gem with version 2.1.3 inside the given project.

  • Parameters:

    description: "..."
tags: [ ... ]
metadata:
  package_type: maven
  checksum: "..."

Example 3: Publish a container image

  • API endpoint: POST catalog/publish/:project_id/images/dast/1.2. This publishes the image dast:1.2 existing inside the given project.

  • Parameters:

    description: "..."
tags: [ ... ]
metadata: { ... }

Metadata

Metadata should be considered optional. It should be possible to publish resources without metadata (e.g. list of components in a project and their inputs) since they are not critical information for the catalog. Such information is just needed to show structured information in the UI.In the case of component inputs, the user would have already:

  • defined spec:inputs as the ultimate SSoT. This spec will raise errors to the consumer if runtime input parameters are invalid.
  • documented these inputs in the README.md. The README is the ultimate documentation for a component project and it should be self sufficient even outside the catalog view.

Immediate actions ⚠

  • Need to hold off from enabling the Components tab in the Catalog details page because it will display metadata about components to the users, locking this UX expectation.
  • Need to hold off from displaying the list of components in a project in the catalog index page, for the same reason above.

We can still do this eventually but for now we can focus on changing the publishing process and then introduce metadata afterwards.

How it affects the current logic

  • Existing logic: Currently the user creates a release and using the release-cli and as part of the release creation we publish the new version to the catalog.
  • New logic: The release-cli by default will only create a release. To publish the new release to the catalog, users have to use the new API.
    • Users will not see new releases in the catalog by default anymore (this is a breaking change).
    • The users will gain control over which versions they want to publish. Maybe there are release candidates that they don't want to show in the catalog.
    • We need to provide tooling for the publishing: either we enhance the release-cli to have a --publish=true option which uses the new API after release is created or we provide a CI component that does the publishing (in the form of template and/or CI step).

Points of debate

  1. CircleCI and GitHub Actions ingest resources on the server-side, using the yaml files as the SSoT for metadata, inputs, outputs, etc. We are instead considering asking users to compile this information and submit it via the Publishing API. Is this the right call since the user has already crafted the spec:inputs very carefully.
    • We still need a limit in the number of components in a project that can be published in a catalog and this limit should not be very high. We need to encourage users to create small groups of cohesive components and not projects containing hundreds of unrelated components.
    • We could still retain the metadata collection on server-side as async process that scans the SSoT files.
  2. Relying on external scan and metadata collection, while being better in offloading this work to the client, it can present maintenance challenges because we won't control anymore what logic the users are using to publish metadata. What if we want to collect new metadata or migrate to a different format? Clients may use various versions of the client-side tool (e.g. release-cli or step) and it will not be easy to change, requiring a breaking change all the time to force users to use the latest version of the tool.
    • If we collect this metadata on the server-side we have full control over the logic and version compatibility.
  3. For component projects, all existing components existing for a given tag should be published at once. Allowing users to select which components to publish can alter the meaning of versioning for repository-based resources such as CI components. We should stick to always publish everything that is available for a given revision since it will anyway be available for usage via the include:component or the step syntax.
Edited by Fabio Pitino