Use historic mode when running scheduled scan execution policies

What does this MR do and why?

This MR updates how we execute scheduled Scan Execution Policy with secret_detection enabled. We are setting

• SECRET_DETECTION_HISTORIC_SCAN to true for the first scan.
• SECRET_DETECTION_LOG_OPTS with a commit range between the last scheduled pipeline and the current one.

This MR is related to issue #391278 (closed).

The use SECRET_DETECTION_HISTORIC_SCAN for the first scan was discussed here. The SECRET_DETECTION_LOG_OPTS was discussed here.

Migration

Up

main: == 20230403190527 AddIndexToSecurityScansOnScanType: migrating ================
main: -- transaction_open?()
main:    -> 0.0000s
main: -- view_exists?(:postgres_partitions)
main:    -> 0.1035s
main: -- index_exists?(:security_scans, [:scan_type, :project_id, :pipeline_id], {:where=>"status = 1", :name=>"index_for_security_scans_scan_type", :algorithm=>:concurrently})
main:    -> 0.0072s
main: -- execute("SET statement_timeout TO 0")
main:    -> 0.0005s
main: -- add_index(:security_scans, [:scan_type, :project_id, :pipeline_id], {:where=>"status = 1", :name=>"index_for_security_scans_scan_type", :algorithm=>:concurrently})
main:    -> 0.0075s
main: -- execute("RESET statement_timeout")
main:    -> 0.0002s
main: == 20230403190527 AddIndexToSecurityScansOnScanType: migrated (0.1453s) =======

New Queries

Get Pipeline ids

SELECT
    "security_scans"."pipeline_id"
FROM
    "security_scans"
WHERE
    "security_scans"."scan_type" = 5
    AND "security_scans"."project_id" = 278964
    AND "security_scans"."status" = 1

Before Index

https://postgres.ai/console/gitlab/gitlab-production-tunnel-pg12/sessions/17241/commands/58024

After Index

https://postgres.ai/console/gitlab/gitlab-production-tunnel-pg12/sessions/17466/commands/58444

New index discussion thread

Get last pipeline sha

SELECT
    "ci_pipelines".*
FROM
    "ci_pipelines"
WHERE
    "ci_pipelines"."project_id" = 278964
    AND "ci_pipelines"."ref" = 'master'
    AND "ci_pipelines"."source" = 15
    AND "ci_pipelines"."id" IN (56, 57, 58, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90)
ORDER BY
    "ci_pipelines"."id" DESC
LIMIT 1

https://postgres.ai/console/gitlab/gitlab-production-tunnel-pg12/sessions/17241/commands/58025

Screenshots

First Scan

Commits

Second Scan

How to set up and validate locally

1. Create a new project.
2. Update the Secret-Detection.gitlab-ci.yml as described here to verify the exported variables.
3. Create a Scan Execution Policy with secret_detection scheduled to the given cadence, and add a variables section to try to disable the SECRET_DETECTION_HISTORIC_SCAN:

type: scan_execution_policy
name: 'Enforce scheduled secret detection '
description: ''
enabled: true
rules:
  - type: schedule
    cadence: '*/5 * * * *'
    branches:
      - '*'
actions:
  - scan: secret_detection
    variables:
      SECRET_DETECTION_HISTORIC_SCAN: 'false'

4. Create a Merge Request with the policy update, and merge it.
5. Wait and observe CI/CD -> Pipelines to see if the first scan was scheduled with SECRET_DETECTION_HISTORIC_SCAN enabled.
6. While waiting for the second scan to run, update the readme file and commit it to main branch.
7. Wait for the second scan to run and observe if it was scheduled with the SECRET_DETECTION_LOG_OPTS with a commit range between the last scheduled pipeline and the current one.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

• I have evaluated the MR acceptance checklist for this MR.