Refactor scan execution policy variable precedence

Why are we doing this work

The enforcement of the scan execution policy variable highest precedence is built on fragile foundation:

  • The performance can be improved because we're rebuilding the SEP configurations in the builder, which happens when pipeline is created and also when a runner picks up a job.
  • We've also encountered bugs with scheduled SEP related to this part of the code: !182889 (closed).

Historical context

Previously, we didn't have a pipeline context and build_source to know which jobs are scan_execution_policy jobs and we had to recalculate the scan execution policy configurations from the builder.

Proposal

We should store the variables defined in scan execution policies in the job.options and enforce them in a similar way as pipeline execution policies. We should refactor it to be more inline with pipeline execution policy precedence enforcement: https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/models/security/pipeline_execution_policy/variables_override.rb.

Relevant links

Implementation plan

  • To be able to use build_source, we need to refactor the step SetBuildSources and assign the build source using nested attributes from YamlProcessor::Result. The reason is that the variables need to be available for Seed step, which happens before SetBuildSources.
  • Using build_source attribute and SEP pipeline context, identify a scan_execution_policy job in YamlProcessor::Result and save the SEP variables directly in job.options
  • In the variables builder, use the variables persisted in job.options and enforce them with the highest precedence without recalculating anything

Rough implementation plan can be found in this thread, however, it's partially outdated: !182889 (comment 2391031014)

Verification steps

Edited by 🤖 GitLab Bot 🤖