Secret Detection - Add flag to scan all commits in MR source branch

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

Problem

A newly discovered secret in an MR disappears when the pipeline runs a second time.

Steps to reproduce

  1. Commit change with a secret
  2. Scan detects secret
  3. Merge is blocked by scan execution policy
  4. Commit change in same file (added a new line, did not remove secret)
  5. Scan runs
  6. Secret vulnerability no longer detected, merge is unblocked

Context

By default, Secret Detection only scans the latest commit to a branch and detects any newly-committed secrets. This creates the problem where a secret disappears from Secret Detection findings when a new pipeline is run. This allows secrets to leak into the main branch. For example, if a project uses a Scan Execution Policy that blocks merges upon secret detection, the policy can be circumvented as described above.

Potential workarounds

The problem no longer occurs after setting SECRET_DETECTION_HISTORIC_SCAN = true in the scan policy details, which will scan all commits in the repository history. However, this can take a long time, and it's recommended in our documentation to do a full history scan only once after enabling Secret Detection, and use only standard Secret Detection after the initial run.

If the customer is not using Scan Execution Policies, they can edit their .gitlab-ci.yml file like this:

secret_detection:
  rules:
    - if: $CI_PIPELINE_SOURCE == 'merge_request_event' # CI_MERGE_REQUEST_SOURCE_BRANCH_NAME variable is only available for MR pipelines
  variables:
    SECRET_DETECTION_LOG_OPTIONS: origin/${CI_MERGE_REQUEST_SOURCE_BRANCH_NAME}..HEAD # Check commits to MR source branch
  before_script:
    - git fetch origin ${CI_MERGE_REQUEST_SOURCE_BRANCH_NAME} # Needed for SECRET_DETECTION_LOG_OPTIONS

However, this does not apply to Scan Execution Policies because custom YAML is not yet supported by policies. Customers using Scan Execution Policies currently have the option to 1) only scan the latest commit, which could lead to leaked secrets or 2) run a full historic scan for each pipeline, which takes a long time.

Proposal

Add a new flag, SECRET_DETECTION_MERGE_REQUEST_SOURCE_BRANCH_SCAN (not set on this name), similar to SECRET_DETECTION_HISTORIC_SCAN, that scans all the commits in current branch. This way, customers using Scan Execution Policies can set SECRET_DETECTION_MERGE_REQUEST_SOURCE_BRANCH_SCAN = true in their policy details so that only the MR source branch is fully scanned for each pipeline in an MR.

Edited by 🤖 GitLab Bot 🤖