Use variables_override concept for SEP

What does this MR do and why?

Use variables_override concept for SEP.

To provide more resilient enforcement of SEP variables precedence, we should switch from recalculating the SEP configs and their variables in the variables builder and use the data we have persisted when the job was created.

Since only the explicitly-listed SEP variables should be enforced with the highest precedence and the rest should keep the original precedence, this concept is the same as when PEP declares variables_override: { allowed: true, exceptions: <my-variables> }.

This change not only improves performance, but also fixes a mix-up of variables when policy contains multiple rule types (schedule + pipeline).

References

• SEP variables incorrectly assigned for multiple... (#485051)
• SEP variables incorrectly overrides for multipl... (#562039 - closed)
• Runner MR to update the wording for logs: Update policy-related logs to be more generic (gitlab-runner!6445 - merged)
• Cleanup SEP variables builder: #591130

Screenshots or screen recordings

Before	After

The logs in the runner needs to be slightly adjusted so that it matches the proposed change. Draft MR: Update policy-related logs to be more generic (gitlab-runner!6445 - merged)

Current	New

How to set up and validate locally

1. Create a project.

2. Add the following CI config (.gitlab-ci.yml):

   # GitLab CI/CD Pipeline Configuration
   
   variables:
     CI_DEBUG_TRACE: true
   
   before_script:
     - echo $GITLAB_ADVANCED_SAST_ENABLED
   
   # Default job for demonstration
   test:
     script:
       - echo "Running tests..."

3. Add test.rb to trigger SAST:

   # WARNING: This class is intentionally insecure for demonstration purposes.
   # Do NOT use in production.
   class UserLookup
     def find_user(username)
       # Vulnerability: passing unsanitized input directly to system()
       system("grep #{username} /etc/passwd")
     end
   end
   
   # Example usage (unsafe):
   # UserLookup.new.find_user("; rm -rf /")

4. Add Gemfile.lock to trigger dependency scanning.

   Example Gemfile.lock

   GEM
     remote: https://rubygems.org/
     specs:
       actioncable (7.0.0)
         actionpack (= 7.0.0)
         activesupport (= 7.0.0)
         nio4r (~> 2.0)
         websocket-driver (>= 0.6.1)
       actionmailbox (7.0.0)
         actionpack (= 7.0.0)
         activejob (= 7.0.0)
         activerecord (= 7.0.0)
         activestorage (= 7.0.0)
         activesupport (= 7.0.0)
         mail (>= 2.7.1)
       actionmailer (7.0.0)
         actionpack (= 7.0.0)
         actionview (= 7.0.0)
         activejob (= 7.0.0)
         activesupport (= 7.0.0)
         mail (~> 2.5, >= 2.5.4)
         rails-dom-testing (~> 2.0)
       actionpack (7.0.0)
         actionview (= 7.0.0)
         activesupport (= 7.0.0)
         rack (~> 2.0, >= 2.2.0)
         rack-test (>= 0.6.3)
         rails-dom-testing (~> 2.0)
         rails-html-sanitizer (~> 1.0, >= 1.2.0)
       actiontext (7.0.0)
         actionpack (= 7.0.0)
         activerecord (= 7.0.0)
         activestorage (= 7.0.0)
         activesupport (= 7.0.0)
         globalid (>= 0.6.0)
         nokogiri (>= 1.8.5)
       actionview (7.0.0)
         activesupport (= 7.0.0)
         builder (~> 3.1)
         erubi (~> 1.4)
         rails-dom-testing (~> 2.0)
         rails-html-sanitizer (~> 1.1, >= 1.2.0)
       activejob (7.0.0)
         activesupport (= 7.0.0)
         globalid (>= 0.3.6)
       activemodel (7.0.0)
         activesupport (= 7.0.0)
       activerecord (7.0.0)
         activemodel (= 7.0.0)
         activesupport (= 7.0.0)
       activestorage (7.0.0)
         actionpack (= 7.0.0)
         activejob (= 7.0.0)
         activerecord (= 7.0.0)
         activesupport (= 7.0.0)
         marcel (~> 1.0)
         mini_mime (>= 1.1.0)
       activesupport (7.0.0)
         concurrent-ruby (~> 1.0, >= 1.0.2)
         i18n (>= 1.6, < 2)
         minitest (>= 5.1)
         tzinfo (~> 2.0)
       base64 (0.3.0)
       builder (3.3.0)
       concurrent-ruby (1.3.5)
       crass (1.0.6)
       date (3.4.1)
       erubi (1.13.1)
       globalid (1.3.0)
         activesupport (>= 6.1)
       i18n (1.14.7)
         concurrent-ruby (~> 1.0)
       loofah (2.24.1)
         crass (~> 1.0.2)
         nokogiri (>= 1.12.0)
       mail (2.8.1)
         mini_mime (>= 0.1.1)
         net-imap
         net-pop
         net-smtp
       marcel (1.1.0)
       method_source (1.1.0)
       mini_mime (1.1.5)
       minitest (5.26.0)
       net-imap (0.5.12)
         date
         net-protocol
       net-pop (0.1.2)
         net-protocol
       net-protocol (0.2.2)
         timeout
       net-smtp (0.5.1)
         net-protocol
       nio4r (2.7.4)
       nokogiri (1.18.10-aarch64-linux-gnu)
         racc (~> 1.4)
       nokogiri (1.18.10-aarch64-linux-musl)
         racc (~> 1.4)
       nokogiri (1.18.10-arm-linux-gnu)
         racc (~> 1.4)
       nokogiri (1.18.10-arm-linux-musl)
         racc (~> 1.4)
       nokogiri (1.18.10-arm64-darwin)
         racc (~> 1.4)
       nokogiri (1.18.10-x86_64-darwin)
         racc (~> 1.4)
       nokogiri (1.18.10-x86_64-linux-gnu)
         racc (~> 1.4)
       nokogiri (1.18.10-x86_64-linux-musl)
         racc (~> 1.4)
       racc (1.8.1)
       rack (2.2.20)
       rack-test (2.2.0)
         rack (>= 1.3)
       rails (7.0.0)
         actioncable (= 7.0.0)
         actionmailbox (= 7.0.0)
         actionmailer (= 7.0.0)
         actionpack (= 7.0.0)
         actiontext (= 7.0.0)
         actionview (= 7.0.0)
         activejob (= 7.0.0)
         activemodel (= 7.0.0)
         activerecord (= 7.0.0)
         activestorage (= 7.0.0)
         activesupport (= 7.0.0)
         bundler (>= 1.15.0)
         railties (= 7.0.0)
       rails-dom-testing (2.3.0)
         activesupport (>= 5.0.0)
         minitest
         nokogiri (>= 1.6)
       rails-html-sanitizer (1.6.2)
         loofah (~> 2.21)
         nokogiri (>= 1.15.7, != 1.16.7, != 1.16.6, != 1.16.5, != 1.16.4, != 1.16.3, != 1.16.2, != 1.16.1, != 1.16.0.rc1, != 1.16.0)
       railties (7.0.0)
         actionpack (= 7.0.0)
         activesupport (= 7.0.0)
         method_source
         rake (>= 12.2)
         thor (~> 1.0)
         zeitwerk (~> 2.5)
       rake (13.3.0)
       thor (1.4.0)
       timeout (0.4.3)
       tzinfo (2.0.6)
         concurrent-ruby (~> 1.0)
       websocket-driver (0.8.0)
         base64
         websocket-extensions (>= 0.1.0)
       websocket-extensions (0.1.5)
       zeitwerk (2.7.3)
   
   PLATFORMS
     aarch64-linux-gnu
     aarch64-linux-musl
     arm-linux-gnu
     arm-linux-musl
     arm64-darwin
     x86_64-darwin
     x86_64-linux-gnu
     x86_64-linux-musl
   
   DEPENDENCIES
     rails (= 7.0.0)
   
   BUNDLED WITH
      2.7.1

5. Create a scheduled SEP:

   scan_execution_policy:
     - name: Scheduled SEP
       description: ''
       enabled: true
       policy_scope:
         projects:
           excluding: []
       actions:
         - scan: sast
           variables:
             SECURE_ENABLE_LOCAL_CONFIGURATION: 'false'
             GITLAB_ADVANCED_SAST_ENABLED: 'true'
           template: latest
       rules:
         - type: schedule
           cadence: 0 0 * * *
           branch_type: all
       skip_ci:
         allowed: true

6. Add a pipeline SEP:

   scan_execution_policy:
     - name: Pipeline SEP
       description: ''
       enabled: true
       policy_scope:
         projects:
           excluding: []
       actions:
         - scan: sast
           variables:
             SECURE_ENABLE_LOCAL_CONFIGURATION: 'false'
             GITLAB_ADVANCED_SAST_ENABLED: 'false'
       rules:
         - type: pipeline
           branch_type: all
       skip_ci:
         allowed: true

7. Add a SEP for dependency scanning:

   scan_execution_policy:
     - name: DS
       description: ''
       enabled: true
       actions:
         - scan: dependency_scanning
           template: latest
       rules:
         - type: pipeline
           branch_type: all
       skip_ci:
         allowed: true

8. Add DS_ENFORCE_NEW_ANALYZER: true in the project's CI/CD settings to test that precedence is not altered for job.yaml_variables, only for the policy variables. With this variable set to true, dependency-scanning-1 should be injected instead of gemnasium-dependency-scanning-1.

9. Add GITLAB_ADVANCED_SAST_ENABLED: true to try to override the policy variable. This shouldn't work and semgrep-sast-0 should be injected.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #485051