Create CI component usage record when component is included in pipeline
What does this MR do and why?
One of our objectives in the parent issue is to enable users to filter/sort CI/CD Catalog components by popularity in the UI. To accomplish this, we introduced a custom table p_catalog_resource_component_usages
to record usage data. The data in this table will later be aggregated and saved into the DB in the next task (#443381 (closed)).
In this MR, we're implementing the logic that inserts the usage data into p_catalog_resource_component_usages
with a new service class Ci::Components::Usages::CreateService
. This logic is placed at the same point where we are sending Internal Events (which was implemented in !146834 (merged).)
We are making this change under the same feature flag that was introduced in !146834 (merged):
ci_track_catalog_component_usage
- Roll out issue: #446290 (closed)
Further context:
While both the new Ci::Components::Usages::CreateService
and Internal Events execute on the same action (i.e. including a catalog component in the pipeline), the data from these two approaches are used for different reasons.
- Internal Events will be used for monitoring data trends on
GitLab.com
(as well as some coarse metrics from self-managed instances). - The data in
p_catalog_resource_component_usages
will be used to allow users to sort/filter components or component projects by popularity (last 30-day usage). This is a more complex task and cannot be easily/efficiently supported by Internal Events (see !144932 (comment 1783721915) for more context).
This MR resolves Child Task #443380 (closed) (Step 2) of Parent Issue #440382 (closed).
MR acceptance checklist
Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
How to set up and validate locally
To test this, we will set up a scenario that involves multiple components to ensure that we're only sending an Internal Event for components that exist in the CI/CD Catalog.
NOTE: In the following steps, replace the <VALUE>
placeholders as needed.
- Enable the feature flag:
Feature.enable(:ci_track_catalog_component_usage)
- Create two new projects
Project 1
andProject 2
(withREADME.md
files) in a group namedGroup C
. Only updateProject 1
to be a CI/CD Catalog project. - In Project 1, create a new
templates
folder and add the following file:
component_a.yml
component-a-job:
script: echo
- In Project 2, create a new
templates
folder and add the following file:
component_x.yml
component-x-job:
script: echo
- For each project:
- Make any change to the
README.md
file and commit it tobranch1
; then commit another change tobranch2
. - Create tags
1.0.0
and2.0.0
for each branch respectively. - Create a release for each tag in the Rails console:
project_1 = Project.find(<PROJECT_1_ID>)
project_2 = Project.find(<PROJECT_2_ID>)
user = User.find(1) # Your root user
[project_1, project_2].each do |project|
project.repository.tags.each do |tag|
Releases::CreateService.new(project, user, tag: tag.name, legacy_catalog_publish: true).execute
end
end
- In Project 2, update your
.gitlab-ci.yml
with the following (this should start a new pipeline run):
include:
- component: gdk.test:3000/group-c/project-1/component_a@main # Not a released/versioned component
- component: gdk.test:3000/group-c/project-1/component_a@1.0.0
- component: gdk.test:3000/group-c/project-1/component_a@2.0.0
- component: gdk.test:3000/group-c/project-2/component_x@1.0.0 # Project is not a catalog resource
- component: gdk.test:3000/group-c/project-2/component_x@2.0.0 # Project is not a catalog resource
With the above pipeline config, we expect only project-1/component_a@1.0.0
and project-1/component_a@2.0.0
to be tracked.
- In the Rails console, check that the correct usage records were inserted into the database:
project_2 = Project.find(<PROJECT_2_ID>)
Ci::Catalog::Resources::Components::Usage.where(used_by_project_id: project_2.id).map do |usage|
{ component_path: "#{usage.project.path}/#{usage.component.name}@#{usage.component.version.name}", used_by_project_name: Project.find(usage.used_by_project_id).name, used_date: usage.used_date }
end
The output should confirm that only two component usages were recorded with the values as shown:
- Run the pipeline in Project 2 again and observe that there are still only two usage records in the table. (They are unique by
component_id
-used_by_project_id
-used_date
.
Related to #440382 (closed)