Enable reading pipeline variables from object storage artifacts

What does this MR do and why?

Enables reading pipeline variables from object storage artifacts instead of the database, behind the feature flag. When ci_read_pipeline_variables_from_artifact feature flag is enabled, pipeline variables are read from encrypted JSON artifacts stored in object storage rather than from the p_ci_pipeline_variables partitions.

Changes:

• Add pipeline_variables method in Gitlab::Ci::Variables::Builder that switches between reading from artifact (when FF enabled) or DB (when FF disabled)
• Add read_pipeline_variables_from_artifact method to parse JSON from pipeline artifact
• Introduced ci_read_pipeline_variables_from_artifact feature flag

How to set up and validate locally

project = Project.first
pipeline = project.ci_pipelines.first

# Create test artifact (if not already created by writes MR)
json_data = [{ key: 'VAR1', value: 'val1' }, { key: 'VAR2', value: 'val2' }].to_json
artifact = Ci::PipelineArtifact.create!(pipeline: pipeline, project: project, file_type: :pipeline_variables, file_format: :raw, file: CarrierWaveStringFile.new_file(file_content: json_data, filename: 'pipeline_variables.json', content_type: 'application/json'), size: json_data.bytesize)

# Read from artifact (FF enabled)
Feature.enable(:ci_read_pipeline_variables_from_artifact)
builder = Gitlab::Ci::Variables::Builder.new(pipeline)
from_artifact = builder.send(:pipeline_variables)

# Read from database (FF disabled)
Feature.disable(:ci_read_pipeline_variables_from_artifact)
builder2 = Gitlab::Ci::Variables::Builder.new(pipeline)
from_db = builder2.send(:pipeline_variables)

{ from_artifact: { type: from_artifact.class, data: from_artifact }, from_db: { type: from_db.class, data: from_db.to_a } }

Results

Related to #580435 (closed)