Add dual-write for pipeline variables artifact to object storage

What does this MR do and why?

Add dual-write for pipeline variables to object storage

When a pipeline completes and the feature flag is enabled, variables are now stored as encrypted artifacts in object storage alongside the existing database storage.

Changes:

  • Add CreateVariablesArtifactService to store variables
  • Add CreateVariablesArtifactWorker triggered on pipeline completion
  • Add has_pipeline_variables_artifact? helper to Ci::Pipeline
  • Add to_artifact_hash method to Ci::PipelineVariable
  • Introduced pipeline_variables_artifact_write feature flag
  • Leverages custom lockbox implementation to handle file type based encryption for uploader
  • Wrote specs

Changelog: added

Object Storage Validation

Setup

# Enable OS in GDK
gdk config set object_store.enabled true
gdk reconfigure
gdk restart

# Verify that MinIO(Object) is running
gdk status

Test with Object Storage in Rails console

# Enable feature flag
Feature.enable(:pipeline_variables_artifact_write)

# Create test data
project = FactoryBot.create(:project, :repository)
pipeline = FactoryBot.create(:ci_pipeline, project: project, status: :success)
FactoryBot.create(:ci_pipeline_variable, pipeline: pipeline, key: 'SECRET', value: 'my-secret-value')

# Create artifact
artifact = Ci::PipelineArtifacts::CreateVariablesArtifactService.new(pipeline).execute

# Verify it's using Object Storage
artifact.file_store
# => 2 (remote/object storage)

# Verify encryption/decryption works
artifact.file.read
# => Returns decrypted JSON: [{"key":"SECRET","value":"my-secret-value","variable_type":"env_var","raw":false}]

# Cleanup
pipeline.destroy
Feature.disable(:pipeline_variables_artifact_write)

Expected Result

Check Expected
artifact.file_store 2 (remote)
artifact.file.read Decrypted JSON

Related to #580107

Edited by Madhusudan Vaishnao

Merge request reports

Loading