Add dual-write for pipeline variables artifact to object storage
What does this MR do and why?
Add dual-write for pipeline variables to object storage
When a pipeline completes and the feature flag is enabled, variables are now stored as encrypted artifacts in object storage alongside the existing database storage.
Changes:
- Add CreateVariablesArtifactService to store variables
- Add CreateVariablesArtifactWorker triggered on pipeline completion
- Add has_pipeline_variables_artifact? helper to Ci::Pipeline
- Add to_artifact_hash method to Ci::PipelineVariable
- Introduced pipeline_variables_artifact_write feature flag
- Leverages custom lockbox implementation to handle file type based encryption for uploader
- Wrote specs
Changelog: added
Object Storage Validation
Setup
# Enable OS in GDK
gdk config set object_store.enabled true
gdk reconfigure
gdk restart
# Verify that MinIO(Object) is running
gdk status
Test with Object Storage in Rails console
# Enable feature flag
Feature.enable(:pipeline_variables_artifact_write)
# Create test data
project = FactoryBot.create(:project, :repository)
pipeline = FactoryBot.create(:ci_pipeline, project: project, status: :success)
FactoryBot.create(:ci_pipeline_variable, pipeline: pipeline, key: 'SECRET', value: 'my-secret-value')
# Create artifact
artifact = Ci::PipelineArtifacts::CreateVariablesArtifactService.new(pipeline).execute
# Verify it's using Object Storage
artifact.file_store
# => 2 (remote/object storage)
# Verify encryption/decryption works
artifact.file.read
# => Returns decrypted JSON: [{"key":"SECRET","value":"my-secret-value","variable_type":"env_var","raw":false}]
# Cleanup
pipeline.destroy
Feature.disable(:pipeline_variables_artifact_write)
Expected Result
| Check | Expected |
|---|---|
artifact.file_store |
2 (remote) |
artifact.file.read |
Decrypted JSON |
Related to #580107
Edited by Madhusudan Vaishnao