Add CI component usage internal event tracking
What does this MR do and why?
Our overall objective in the parent issue #440382 (closed) is to implement component usage instrumentation. Specifically, we want to track only components that exist in the CI/CD Catalog (i.e. ones that have corresponding records in the catalog_resource_components
table.) This MR resolves the requirement to monitor usage trends of components on GitLab.com
, which we can accomplish using Internal Events.
In !147378 (merged), we introduced two methods:
-
included_components
: returns theproject
,name
, andsha
of any component included in the pipeline using theinclude:component
keyword. -
find_catalog_component
: only returns components that exist incatalog_resource_components
.
In this MR, we now utilize the two methods in a new pipeline creation chain step ComponentUsage
. For each catalog component found, we send an Internal Event. This will allow us to monitor the usage trends of components hosted on Gitlab.com
.
Internal event note:
- We're making use of additional properties so that we can later break down the data by component name (and version) and/or component project path.
Feature flag:
ci_track_catalog_component_usage
- Roll out issue: #446290 (closed)
Resolves Child Task #443382 (closed) of Parent Issue #440382 (closed).
How to set up and validate locally
To test this, we will set up a scenario that involves multiple components to ensure that we're only sending an Internal Event for components that exist in the CI/CD Catalog.
NOTE: In the following steps, replace the <VALUE>
placeholders as needed.
- Enable the feature flag:
Feature.enable(:ci_track_catalog_component_usage)
- Set up your local gdk to run Snowplow: Setup local event collector
- Start the Internal Events Monitor:
rails runner scripts/internal_events/monitor.rb ci_catalog_component_included
- Create two new projects
Project 1
andProject 2
(withREADME.md
files) in a group namedGroup C
. Only updateProject 1
to be a CI/CD Catalog project. - In Project 1, create a new
templates
folder and add the following file:
component_a.yml
component-a-job:
script: echo
- In Project 2, create a new
templates
folder and add the following file:
component_x.yml
component-x-job:
script: echo
- For each project:
- Make any change to the
README.md
file and commit it tobranch1
; then commit another change tobranch2
. - Create tags
1.0.0
and2.0.0
for each branch respectively. - Create a release for each tag in the Rails console:
project_1 = Project.find(<PROJECT_1_ID>)
project_2 = Project.find(<PROJECT_2_ID>)
user = User.find(1) # Your root user
[project_1, project_2].each do |project|
project.repository.tags.each do |tag|
Releases::CreateService.new(project, user, tag: tag.name).execute
end
end
- In any project, update your
.gitlab-ci.yml
with the following (this should start a new pipeline run):
include:
- component: gdk.test:3000/group-c/project-1/component_a@main # Not a released/versioned component
- component: gdk.test:3000/group-c/project-1/component_a@1.0.0
- component: gdk.test:3000/group-c/project-1/component_a@2.0.0
- component: gdk.test:3000/group-c/project-2/component_x@1.0.0 # Project is not a catalog resource
- component: gdk.test:3000/group-c/project-2/component_x@2.0.0 # Project is not a catalog resource
With the above pipeline config, we expect only component_a@1.0.0
and component_a@2.0.0
to be tracked.
- In the Internal Event Monitor, enter 'p' to refresh the data. Observe that we have exactly 2 Snowplow events recorded. The Redis HLL counters should be 1 since they're unique by project_id. The
project_id
should be of the project that ran the pipeline.
- For good measure, let's also make sure the additional properties of the Snowplow events are correct.
- Open the local Snowplow Micro UI by going to
http://localhost:9091/micro/ui
. - In the Events Table section, search
ci_catalog_component_included
. - Open the Event Details of the last two events.
- Expand root > rawEvent > parameters.
- Observe that
se_va
=1
andse_la
=group-c/project-1
for both.se_pr
should becomponent_a@2.0.0
for one andcomponent_a@1.0.0
for the other.
MR acceptance checklist
Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
Related to #440382 (closed)