Extract the CI catalog components included a pipeline
Background context
When a version (aka release) is created for a catalog resource (aka a "published component project" or "CI/CD Catalog project"), a metadata record is created in catalog_resource_components
for each valid component file for the given version. We insert the version_id
and name
for each record to identify the corresponding component file.
All records in catalog_resource_components
represent a valid component in the CI/CD Catalog. However, it's possible to have components that are not in the CI/CD catalog, so here we will use these terms to distinguish them:
-
catalog component: Refers to a component in the CI/CD catalog (i.e. one that has a corresponding record in
catalog_resource_components
). - component: Refers to any component.
What does this MR do and why?
Our overall objective in the parent issue #440382 (closed) is to implement component usage instrumentation for both GitLab.com
and Self-Managed instances. As part of this objective, we must determine when a component is "used", which is defined as below:
When a CI/CD Catalog component is included in a pipeline with the
include:component
keyword.
To track this usage, we will implement a process that's similar to how included_templates are collected and then used later in Ci::Pipeline::Chain::TemplateUsage.
In this MR, we implement the first part of the process: extraction of included_components
. The basic idea is that we save the metadata of each component (project
, sha
, and name
) in an array.
In the next step, the resulting YamlProcessor::Result#included_components
will be used in a new pipeline creation chain process where we will evoke the find_catalog_component
method.
Partially resolves Child Task Step 2 of Parent Issue #440382 (closed).
How to set up and validate locally
To test this, we will set up a scenario that involves multiple components to ensure that Ci::YamlProcessor::Result#included_components
produces the correct result.
NOTE: In the following steps, replace the <VALUE>
placeholders as needed.
- Create two new projects
Project 1
andProject 2
(withREADME.md
files) in a group namedGroup C
. - In Project 1, create a new
templates
folder and add the following files.
component_a.yml
component-a-job:
script: echo
component_b.yml
component-b-job:
script: echo
component_c.yml
include:
- component: gdk.test:3000/group-c/project-1/component_a@2.0.0
component-c-job:
script: echo
- In Project 2, create a new
templates
folder and add the following file:
component_x.yml
component-x-job:
script: echo
- For each project:
- Make any change to the
README.md
file and commit it tobranch1
; then commit another change tobranch2
. - Create tags
1.0.0
and2.0.0
for each branch respectively.
- Ensure the following Yaml is valid in the Pipeline editor. This is just to make sure that everything is set up properly.
include:
- component: gdk.test:3000/group-c/project-1/component_a@1.0.0
- component: gdk.test:3000/group-c/project-1/component_a@2.0.0
- component: gdk.test:3000/group-c/project-1/component_b@1.0.0
- component: gdk.test:3000/group-c/project-1/component_b@2.0.0
- component: gdk.test:3000/group-c/project-1/component_c@1.0.0
- component: gdk.test:3000/group-c/project-1/component_c@2.0.0
- component: gdk.test:3000/group-c/project-2/component_x@1.0.0
- component: gdk.test:3000/group-c/project-2/component_x@2.0.0
- Next we will pass in the following pipeline yaml into
Gitlab::Ci::YamlProcessor
and observe the result.
include:
- project: group-c/project-1
file: templates/component_a.yml # Excluded because it's not using `include:component`
ref: 1.0.0
- component: gdk.test:3000/group-c/project-1/component_b@main
- component: gdk.test:3000/group-c/project-1/component_b@2.0.0
- component: gdk.test:3000/group-c/project-1/component_b@2.0.0 # Duplicate
- component: gdk.test:3000/group-c/project-1/component_c@1.0.0 # Includes nested component_a@2.0.0
- component: gdk.test:3000/group-c/project-2/component_x@2.0.0
With the above yaml, we expect included_catalog_components
to contain metadata for all unique components except project-1/component_a@1.0.0
.
Run the following in the rails console:
project = Project.find(<PROJECT_1_ID>)
current_user = User.find(1) # Root user
yaml =
<<~YAML
include:
- project: group-c/project-1
file: templates/component_a.yml
ref: 1.0.0
- component: gdk.test:3000/group-c/project-1/component_b@main
- component: gdk.test:3000/group-c/project-1/component_b@2.0.0
- component: gdk.test:3000/group-c/project-1/component_b@2.0.0
- component: gdk.test:3000/group-c/project-1/component_c@1.0.0
- component: gdk.test:3000/group-c/project-2/component_x@2.0.0
YAML
result = Gitlab::Ci::YamlProcessor.new(yaml, project: project, user: current_user).execute
result.included_components.map do |c|
project = c[:component_project]
sha = c[:component_sha]
name = c[:component_name]
"#{project.path}/#{name}@#{project.repository.tag_names_contains(sha, limit: 1).first || 'main'}"
end
And we observe that the output is as expected:
MR acceptance checklist
Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
Related to #440382 (closed)