Integrate SD Exclusions with Scan Diffs feature

What does this MR do and why?

This merge request integrates SD Exclusions feature, specifically the code handling exclusions in the secrets push check introduced in !166511 (merged) with the scan only committed changes (diffs) from !162782 (merged) and !166055 (merged). The aim here is to keep parity between both scanning mechanisms (i.e. old scanning of entire files, and diff scanning which is introduced recently behind a feature flag).

Resolves #497992 (closed).

MR acceptance checklist

I have evaluated this MR against the MR acceptance checklist.

Screenshots or screen recordings

NOTE: All tokens used in the screen recording demo below are made up dummy tokens that was only used for illustrative purposes.

secret-detection-exclusions-merge-request-demo

How to set up and validate locally

• To test the exclusions work as expected, select a project that exist already in your GDK or create a new one.
• Find the project id for the project you have chosen or just created.
• Make sure secret push protection is enabled for the project.
  • Enable pre_receive_secret_detection_push_check FF for the project from Rails console (if not enabled already).
• Ensure to enable the secret_detection_project_level_exclusions FF for the project from Rails console.

> project = Project.find(PROJECT_ID)

> Feature.enable(:secret_detection_project_level_exclusions, project)

• Add a number of new files in the projects, make sure to use a different token for each, and commit them:
  • nested/path/nested-path-exclusion.txt – dummy token you may use: glpat-1234567890123456ABCD
  • path-exclusion.txt – dummy token you may: glpat-12345678900123456789
  • raw-value-exclusion.txt – dummy token you may use: glpat-ABCD1234567890123456
  • rule-exclusion.txt – dummy token you may use: glrt-12345678900123456789
• After commiting the files, try to push to the repository, validate that the push fails with error message highlighting found secrets.
• Navigate to secret detection configuration page: PROJECT_PATH/-/security/configuration/secret_detection.
• Add a new exclusion to match each one of the files based on their types:
  • nested/path/nested-path-exclusion.txt – use path type with value: nested/**/*.txt.
  • path-exclusion.txt – use path type with value: path-exclusion.txt.
  • raw-value-exclusion.txt – use raw value type with value defined as the same dummy token you've used.
  • rule-exclusion.txt– useruletype withvalue` defined as the same rule you have chosen to exclude.
• Try to push again. Validate the push is successful and no error message is displayed.