Introduce PipelineVariablesArtifactBuilder to store pipeline variables into object storage
What does this MR do and why?
Context
Currently pipeline variables take up a lot of space in the database table p_ci_pipeline_variables. Our goal in the epic gitlab-org#19989 is to offload this data to object storage. To do this, we will leverage our existing pipeline artifacts framework to store the pipeline variables as JSON in an artifact.
We have an approved POC MR (!218525 (diffs)) that contains the overall approach for this objective.
This MR
This MR is the next step of the "write to object storage" component, based on the POC MR.
In !219687 (merged), we introduced the Ci::PipelineVariableItem virtual model to represent variables handled in the object-storage context. In !220489 (merged), we added the new pipeline artifact file_type :pipeline_variables and enabled PipelineArtifactUploader to encrypt/decrypt pipeline_variables files.
Now in this MR, we introduce PipelineVariablesArtifactBuilder which encapsulates the logic that accepts variables_attributes (array of hashes) and transforms the data into a pipeline_variables artifact. It ensures that the variable attributes are validated before building the artifact.
This builder class will later be hooked in to the pipeline creation step Ci::Pipeline::Chain::Build::Associations, as part of the next task: Build pipeline variables artifact behind FF in ... (#586935 - closed).
References
- This MR resolves the task: Introduce PipelineVariablesArtifactBuilder class (#586934 - closed)
- Next task: Build pipeline variables artifact behind FF in ... (#586935 - closed)
- Parent issue: Store pipeline variables as encrypted pipeline_... (#580107 - closed)
- POC MR: Draft: POC Write and read pipeline variables fr... (!218525 - closed)
How to set up and validate locally
- In the Rails console, you can confirm that the class successfully builds a new pipeline variables artifact.
project = Project.last
old_pipeline = Ci::Pipeline.last # Just a pipeline to copy data from
variables_attributes = [{ key: 'MY_KEY', value: 'MY_VALUE' }]
# Build a new pipeline (to confirm it persists the pipeline artifact later)
pipeline = Ci::Pipeline.new(project: project, ref: old_pipeline.ref, sha: old_pipeline.sha, source: old_pipeline.source)
# Run the builder
Gitlab::Ci::Pipeline::Build::PipelineVariablesArtifactBuilder.new(pipeline, variables_attributes).run
# Verify that there is a pipeline variables artifact and its contents are expected
expected_variables_attributes = [{ key: 'MY_KEY', value: 'MY_VALUE', variable_type: 'env_var', raw: false }.stringify_keys]
Gitlab::Json.parse(pipeline.pipeline_artifacts_pipeline_variables.file.read) == expected_variables_attributes # Should return true
# Persist the pipeline
pipeline.save!
# Confirm that the pipeline_variables artifact is also persisted.
pipeline.pipeline_artifacts_pipeline_variables.persisted? # Should return trueMR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
Related to #586934 (closed)