Build pipeline variables artifact in Chain::Build::Associations behind ci_write_pipeline_variables_artifact

What does this MR do and why?

Context

As part of epic gitlab-org#19989, we are working to reduce database storage by offloading pipeline variables from the p_ci_pipeline_variables table to object storage. This will be done by leveraging the existing pipeline artifacts framework to store the variables as encrypted JSON artifacts.

This MR contains changes that were referenced and taken from the dual writes MR (!215514 (closed)) and the POC MR (!218525 (closed)).

Quick Recap

Previously -

In !219687 (merged), we introduced the Ci::PipelineVariableItem virtual model to represent variables handled in the object-storage context.

In !220489 (merged), we added the new pipeline artifact file_type :pipeline_variables and enabled PipelineArtifactUploader to encrypt/decrypt pipeline_variables files.

In !221190 (merged) we introduce PipelineVariablesArtifactBuilder which encapsulates the logic that accepts variables_attributes (array of hashes) and transforms the data into a pipeline_variables artifact.

This MR

This MR is the next step of the "write to object storage" component.

It hooks up the PipelineVariablesArtifactBuilder (introduced in !221190 (merged)) into the pipeline creation chain step Chain::Build::Associations behind the feature flag ci_write_pipeline_variables_artifact. This change enables us to start dual-writing pipeline variables to both the database and object storage, which is a critical step toward eventually migrating away from database storage.

This MR also:

• Adds logic to catch and handle validation errors during artifact building
• Updates GitLab::Chat::Command to use variables_attributes: with CreatePipelineService instead of building the pipeline variables via seed_block. This ensures that ChatOps-created pipelines (with predefined variables like CHAT_INPUT, CHAT_CHANNEL, CHAT_USER_ID) are also persisted to object storage through the same variable_attributes flow, maintaining consistency across all pipeline creation paths.

References

This MR resolves the task: Build pipeline variables artifact behind FF in ... (#586935 - closed)

Previous task: Introduce PipelineVariablesArtifactBuilder class (#586934 - closed)

Parent issue: Store pipeline variables as encrypted pipeline_... (#580107 - closed)

How to set up and validate locally

UI Testing Setup -

1. Enable the feature flag in rails console:

Feature.enable(:ci_write_pipeline_variables_artifact)

2. Create a new pipeline with a couple pipeline variables set.

   • Go to Build --> Pipelines --> Run pipeline
   • Add variables:
     • Variable: PIPELINE_ENV_VAR = my pipeline environment variable
     • File: PIPELINE_FILE_VAR = my pipeline file variable
   • Click New pipeline

3. In the Rails console, query the pipeline that was last created.

   pipeline = Ci::Pipeline.last
   
   # Verify variables are stored in the artifact
   pipeline.pipeline_artifacts_pipeline_variables.file_type
   # => "pipeline_variables"
   
   ## Verify variables are ALSO stored in the database (dual-write)
   pipeline.variables.map { |var| var.attributes.slice('key', 'value', 'variable_type') }
   # => [{"key"=>"PIPELINE_ENV_VAR", "value"=>"my pipeline environment variable", "variable_type"=>"env_var"},
   # {"key"=>"PIPELINE_ENV_VAR2", "value"=>"secondary pipeline variable", "variable_type"=>"env_var"},
   # {"key"=>"PIPELINE_FILE_VAR", "value"=>"pipeline variable file type", "variable_type"=>"file"}]
   artifact = pipeline.pipeline_artifacts_pipeline_variables
   artifact.file_store
   # => 2 (remote/object storage)
   artifact.file.read
     Decrypt File (0.3ms) Decrypted file
   => "[{\"key\":\"PIPELINE_ENV_VAR1\",\"value\":\"This is a new pipeline environment variable \",\"variable_type\":\"env_var\",\"raw\":false},{\"key\":\"PIPELINE_ENV_VAR3\",\"value\":\"This is another pipeline environment variable \",\"variable_type\":\"env_var\",\"raw\":false},{\"key\":\"PIPELINE_FILE_VAR1\",\"value\":\"This is a file which stores the pipeline variables\",\"variable_type\":\"file\",\"raw\":false}]"

Result -

File store being Object storage

Rails console Testing Setup

# Enable OS in GDK
gdk config set object_store.enabled true
gdk reconfigure
gdk restart

# Verify that MinIO(Object) is running
gdk status

Test with Object Storage in Rails console

# 1. Enable feature flag
Feature.enable(:ci_write_pipeline_variables_artifact)

# 2. Setup test data
project = Project.last
user = User.last

# 3. Create command with variables
vars = [{ key: 'MY_SECRET', value: 'super-secret-value' }, { key: 'MY_FILE_VAR', value: 'file-content', variable_type: 'file' }]
opts = { source: :web, origin_ref: project.default_branch, checkout_sha: project.commit.id }
opts.merge!(project: project, current_user: user, variables_attributes: vars)
command = Gitlab::Ci::Pipeline::Chain::Command.new(**opts)

# 4. Create pipeline and run Build::Associations step
pipeline = Ci::Pipeline.new(project: project, ref: project.default_branch, sha: project.commit.id, source: :web);step = Gitlab::Ci::Pipeline::Chain::Build::Associations.new(pipeline, command);step.perform!

# 5. Verify artifact is built in memory (before save)
pipeline.pipeline_artifacts_pipeline_variables.present?
# => true
pipeline.pipeline_artifacts_pipeline_variables.file_type
# => "pipeline_variables"
pipeline.pipeline_artifacts_pipeline_variables.persisted?
# => false

# 6. Verify variables are also assigned to pipeline (dual-write)
pipeline.variables.map(&:key)
# => ["MY_SECRET", "MY_FILE_VAR"]

# 7. Save pipeline (persists artifact to object storage)
pipeline.save!

# 8. Verify artifact is persisted to Object Storage
artifact = pipeline.pipeline_artifacts_pipeline_variables
artifact.persisted?
# => true
artifact.file_store
# => 2 (remote/object storage)

# 9. Verify encryption/decryption works
content = Gitlab::Json.parse(artifact.file.read)
content.map { |v| v['key'] }
# => ["MY_SECRET", "MY_FILE_VAR"]

# 10. Cleanup
pipeline.destroy
Feature.disable(:ci_write_pipeline_variables_artifact)

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #586935 (closed)