Corrective Action: Calling Ci::Config outside of Ci::YamlProcessor causes problems

Background

• Incident: gitlab-com/gl-infra/production#17819 (closed)
• Sentry: https://new-sentry.gitlab.net/organizations/gitlab/issues/759784/?project=3
• An actual exception example that caused the incident: https://log.gprd.gitlab.net/app/discover#/doc/7092c4e2-4eb5-46f2-8305-a7da2edad090/pubsub-rails-inf-gprd-020813?id=5uRSzY4B9bGOSFMa-_e5

Security::SecurityOrchestrationPolicies::CiAction::Template calls an internal CI class without setting the proper context:

        def scan_template(scan_type)
          template = ::TemplateFinder.build(:gitlab_ci_ymls, nil, name: SCAN_TEMPLATES[scan_type]).execute
          Gitlab::Ci::Config.new(template.content).to_hash
        end

Summary

• When a Runner asks for a job from GitLab, Gitlab::Ci::Variables::Builder::ScanExecutionPolicies is called to calculate the variables.
• And it calls Security::SecurityOrchestrationPolicies::ScanPipelineService.
• And it calls Gitlab::Ci::Config. That's the first problem we have. ❗

Since Gitlab::Ci::Config is called outside of the YamlProcessor scope, YamlProcessor::FeatureFlags raises an error. Normally, we catch this error and send it to Sentry without raising on production. However, since there are too many errors, we pushed the limit of Sentry. And that caused the actual incident.

Other possible places that can cause problems

ee/app/services/security/security_orchestration_policies/scan_pipeline_service.rb:
  66        def collect_config_variables(actions, configs)
  67          actions.zip(configs).each_with_object({}) do |(action, config), hash|
  68            variables = scan_variables_with_action_variables(action, fallback: action_variables(action))
  69:           jobs = custom_scan?(action) ? Gitlab::Ci::Config.new(config.to_yaml).jobs : config
  70  
  71            jobs&.each_key do |key|
  72              hash[key] = variables

ee/app/services/security/security_orchestration_policies/ci_action/custom.rb:
  5      module CiAction
  6        class Custom < Base
  7          def config
  8:           @ci_config = Gitlab::Ci::Config.new(yaml_config, inject_edge_stages: false, user: @context.user)
  9  
  10            job_names = parse_job_names
  11  

ee/app/services/security/security_orchestration_policies/ci_action/template.rb:
  36  
  37          def scan_template(scan_type)
  38            template = ::TemplateFinder.build(:gitlab_ci_ymls, nil, name: SCAN_TEMPLATES[scan_type]).execute
  39:           Gitlab::Ci::Config.new(template.content).to_hash
  40          end
  41  
  42          def hidden_job?(job_name)