Allow to perform repository scan along with commit range scan in Secret Detection scanner
Why are we doing this work
We would like to add an option (ie. SECRET_DETECTION_INCLUDE_REPOSITORY_SCAN
) to the Secret Detection scanner to perform both scans (Default branch
along with Commit range
). With this ability in the scanner, we would be able to properly assess risks related to leaked tokens in the repository as we want to ensure that tokens were not leaked in previous commits and are not leaked in the current state of the repository.
Without this and with the usage of Secret Detection in Scheduled Scan Execution Policies, this scenario is possible:
- We have a project where we configure Secret Detection with the scheduled Scan Execution Policy. In one file, we have a leaked token. The initial commit has sha:
INITIAL_SHA
- We await the first pipeline with the scheduled Secret Detection run. Initially,
SECRET_DETECTION_HISTORIC_SCAN
is set to true - We introduce commit (with sha:
COMMIT_SHA
), where we change other files, but we leave the token unchanged, so the vulnerability is still there. - The second pipeline with scheduled Secret Detection is executed,
SECRET_DETECTION_LOG_OPTIONS
is set toINITIAL_SHA..COMMIT_SHA
, so we are scanning only the last commit, which contains no changes to the file with the leaked token. Based on that, we mark the vulnerability found in the first scheduled run as resolved, although the leak is still in the repository.
In the scope of this issue, we would like to also ensure this newly added option is used by scheduled Scan Execution Policies (https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/services/security/security_orchestration_policies/create_pipeline_service.rb#L72)
Relevant links
Non-functional requirements
-
Documentation: new option is documented in the documentation, - [-] Feature flag:
- [-] Performance:
-
Testing:
Implementation plan
-
groupstatic analysis MR1: add new option to https://gitlab.com/gitlab-org/security-products/analyzers/secrets SECRET_DETECTION_INCLUDE_REPOSITORY_SCAN
to perform both scans at the same time, -
groupsecurity policies MR2: extend ee/app/services/security/security_orchestration_policies/create_pipeline_service.rb
to set new optionSECRET_DETECTION_INCLUDE_REPOSITORY_SCAN: true
along withSECRET_DETECTION_LOG_OPTIONS
,
Verification steps
- Ensure you are on GitLab Ultimate group.
- Create a new project, and add a new file with the exemplary leaked token.
- Create a new Scan Execution Policy with a scheduled Secret Detection scan.
name: Secret Detection Scheduled
description: ''
enabled: true
actions:
- scan: secret_detection
rules:
- type: schedule
cadence: "*/15 * * * *"
branch_type: default
- Wait for the first run of the scheduled scan. Look in the Vulnerability report if the token is detected.
- Create a new commit, but do not change the file with the leaked token.
- Wait for the second run of the scheduled scan.
- Go to the Vulnerability report and ensure that the vulnerability is not marked as remediated.