Skip to content

Invoke secret detection scanning logic from gem

Ahmed Hemdan requested to merge invoke-scanning-logic-from-gem into master

What does this MR do and why?

This merge request is updates the secrets push check introduced over a number of earlier merge requests:

To perform secret detection scans on git blobs using the gitlab-secret_detection gem introduced in !136381 (merged).

The git blobs are loaded using ListAllBlobs() (or alternatively ListBlobs when no quarantine directory exist), and then blobs that already exist in the repository are filtered out, and any binary blobs are rejected. The gem is then invoked with the blobs to scan, and the results are presented to the user.

Note: the merge request is part of a list of related merge requests, which were created iteratively to ensure ease of review and focused scope. Therefore, it shouldn't be reviewed in isolation from the rest of those merge requests.

Resolves #427046 (closed), and built on top of !136896 (merged) and !136381 (merged).

Related Merge Requests

Step Merge Request Description
5 !136896 (merged) Updates the check to list and filter blobs of new git-push operations.
6 This one. Invokes gitlab-secret_detection gem to scan blobs filtered.
7 !138790 (merged) Updates the check to add details of secrets detected (e.g. file path/commit sha)
8 !138831 (merged) Updates the check to introduce a bypass mechanism via commit special flag.

Screenshots or screen recordings

Here's a screenshot of the output received when a secret is detected. We currently only show the blob_id, line_number (relative to the blob), type of secret, and description. Later on, as we continue to work on #427047 (closed), we should likely be able to display other data such as file path and commit sha for which the secret belongs.

Screenshot_2023-11-30_at_9.31.40_PM

How to set up and validate locally

  • Create a new project or using an existing one.
  • In your rails console, execute the following commands:
pry(main)> Gitlab::CurrentSettings.update!(pre_receive_secret_detection_enabled: true) # to ensure feature is enabled instance-wide

pry(main)> project = Project.find(PROJECT_ID)
pry(main)> Feature.enable(:pre_receive_secret_detection_push_check, project)
  • Ensure your GDK is licensed as ultimate.
  • In your terminal, navigate to the project folder.
  • Create a new file, e.g. .env, and add a gitlab personal access token:
TOKEN=glpat-JUST20LETTERSANDNUMB
  • Run git add . and git commit -m 'test' to commit the file.
  • Run the command to push the commited file git push.
  • Verify the push fails due to the secret detected.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Ahmed Hemdan

Merge request reports