Skip to content

Create CI component usage record when component is included in pipeline

Leaminn Ma requested to merge ci-track-and-create-component-usage into master

What does this MR do and why?

One of our objectives in the parent issue is to enable users to filter/sort CI/CD Catalog components by popularity in the UI. To accomplish this, we introduced a custom table p_catalog_resource_component_usages to record usage data. The data in this table will later be aggregated and saved into the DB in the next task (#443381 (closed)).

In this MR, we're implementing the logic that inserts the usage data into p_catalog_resource_component_usages with a new service class Ci::Components::Usages::CreateService. This logic is placed at the same point where we are sending Internal Events (which was implemented in !146834 (merged).)

We are making this change under the same feature flag that was introduced in !146834 (merged):

Further context:

While both the new Ci::Components::Usages::CreateService and Internal Events execute on the same action (i.e. including a catalog component in the pipeline), the data from these two approaches are used for different reasons.

  • Internal Events will be used for monitoring data trends on GitLab.com (as well as some coarse metrics from self-managed instances).
  • The data in p_catalog_resource_component_usages will be used to allow users to sort/filter components or component projects by popularity (last 30-day usage). This is a more complex task and cannot be easily/efficiently supported by Internal Events (see !144932 (comment 1783721915) for more context).

This MR resolves Child Task #443380 (closed) (Step 2) of Parent Issue #440382 (closed).

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

How to set up and validate locally

To test this, we will set up a scenario that involves multiple components to ensure that we're only sending an Internal Event for components that exist in the CI/CD Catalog.

NOTE: In the following steps, replace the <VALUE> placeholders as needed.

  1. Enable the feature flag: Feature.enable(:ci_track_catalog_component_usage)
  2. Create two new projects Project 1 and Project 2 (with README.md files) in a group named Group C. Only update Project 1 to be a CI/CD Catalog project.
  3. In Project 1, create a new templates folder and add the following file:

component_a.yml

component-a-job:
  script: echo
  1. In Project 2, create a new templates folder and add the following file:

component_x.yml

component-x-job:
  script: echo
  1. For each project:
  • Make any change to the README.md file and commit it to branch1; then commit another change to branch2.
  • Create tags 1.0.0 and 2.0.0 for each branch respectively.
  • Create a release for each tag in the Rails console:
project_1 = Project.find(<PROJECT_1_ID>)
project_2 = Project.find(<PROJECT_2_ID>)
user = User.find(1) # Your root user

[project_1, project_2].each do |project|
  project.repository.tags.each do |tag|
    Releases::CreateService.new(project, user, tag: tag.name, legacy_catalog_publish: true).execute
  end
end
  1. In Project 2, update your .gitlab-ci.yml with the following (this should start a new pipeline run):
include:
  - component: gdk.test:3000/group-c/project-1/component_a@main # Not a released/versioned component
  - component: gdk.test:3000/group-c/project-1/component_a@1.0.0
  - component: gdk.test:3000/group-c/project-1/component_a@2.0.0
  - component: gdk.test:3000/group-c/project-2/component_x@1.0.0 # Project is not a catalog resource
  - component: gdk.test:3000/group-c/project-2/component_x@2.0.0 # Project is not a catalog resource

With the above pipeline config, we expect only project-1/component_a@1.0.0 and project-1/component_a@2.0.0 to be tracked.

  1. In the Rails console, check that the correct usage records were inserted into the database:
project_2 = Project.find(<PROJECT_2_ID>)

Ci::Catalog::Resources::Components::Usage.where(used_by_project_id: project_2.id).map do |usage|
  { component_path: "#{usage.project.path}/#{usage.component.name}@#{usage.component.version.name}", used_by_project_name: Project.find(usage.used_by_project_id).name, used_date: usage.used_date }
end

The output should confirm that only two component usages were recorded with the values as shown:

Screenshot_2024-03-28_at_5.03.49_PM

  1. Run the pipeline in Project 2 again and observe that there are still only two usage records in the table. (They are unique by component_id-used_by_project_id-used_date.

Related to #440382 (closed)

Edited by Leaminn Ma

Merge request reports