Secret Detection should scan all commits on the current branch

Release notes

Secret detection now checks for all commits within a merge request instead of just the most recent commit.

Problem to solve

Secret Detection only scans the last commit. If previous commits contain secrets, they will not be detected. Merge-requests without the squash option will be merged into the main branch.

The option SECRET_DETECTION_HISTORIC_SCAN on the other hand allows you to scan all commits, but "can take a long time, especially for larger repositories with lengthy Git histories."

Proposal

Secret detection should scan all commits in a merge request by default.

Workaround

# Secret-Detection for complete branch
secret_detection:
  variables:
    SECRET_DETECTION_LOG_OPTIONS: origin/${CI_DEFAULT_BRANCH}..HEAD # Check commits to default branch, e.g. main
  before_script:
    - git fetch origin ${CI_DEFAULT_BRANCH} # Needed for SECRET_DETECTION_LOG_OPTIONS

Will only check up to 50 commits, can be increased by setting GIT_DEPTH in variables to a higher value.

Drawback: Will check commits up to the default branch, not to the parent branch. So for example for a branch B of a branch A of main the check within branch B will not only check commits of branch B, but also check commits of branch A (up to GIT_DEPTH commits).

Intended users

• Sam (Security Analyst)
• Sasha (Software Developer)

Technical requirements

1. To support this issue, the new secret detection template needs to be run in a merge request pipeline, please see MR pipeline support in Sec templates (#410880 - closed). This work was complete in Enable Secret Detection MR pipelines (#372262 - closed) for groupsecret detection. To confirm: scanners can mix different types of pipelines?
2. CI/CD templates was deprecated in %17.0 and will be removed in 19.0 as a breaking change. CI/CD templates will be replaced with CI/CD Components. As such, we will need to Prepare secret-detection CI component for GA (#413878). The secret detection CI/CD component will need to support merge request pipelines too. Also of note: Update Security template guidance to prefer com... (#489904).