Skip to content

Display blob metadata for found secrets in secrets push check

Ahmed Hemdan requested to merge display-blob-metadata-for-secrets into master

What does this MR do and why?

This merge request updates the secrets push check introduced over a number of earlier merge requests:

To perform secret detection scans on git blobs using the gitlab-secret_detection gem introduced in !136381 (merged).

In this merge request, we update the check to load up a blob metadata using GetTreeEntries() gitaly RPC by listing tree entries for the new revisions (from newrev in changes_access.changes) pushed in a specific push, then cross referencing those entries with the blobs in which a secret is found, allowing us to display the file path and the commit sha for a certain secret to the user.

Note: the merge request is part of a list of related merge requests, which were created iteratively to ensure ease of review and focused scope. Therefore, it shouldn't be reviewed in isolation from the rest of those merge requests.

Resolves #427047 (closed), and built on top of !136896 (merged), !136381 (merged), and !137812 (merged).

Related Merge Requests

Step Merge Request Description
5 !136896 (merged) Updates the check to list and filter blobs of new git-push operations.
6 !137812 (merged) Invokes gitlab-secret_detection gem to scan blobs filtered.
7 This one. Updates the check to add details of secrets detected (e.g. file path/commit sha).
8 !138831 (merged) Updates the check to introduce a bypass mechanism via commit special flag.

Screenshots or screen recordings

Before After
image Screenshot_2023-12-07_at_3.24.49_PM

How to set up and validate locally

  • Create a new project or use an existing one.
  • In your rails console, execute the following commands:
pry(main)> Gitlab::CurrentSettings.update!(pre_receive_secret_detection_enabled: true) # to ensure feature is enabled instance-wide

pry(main)> project = Project.find(PROJECT_ID)
pry(main)> Feature.enable(:pre_receive_secret_detection_push_check, project) 
  • Ensure your GDK is licensed as ultimate.
  • In your terminal, navigate to the project folder.
  • Create a new file, e.g. .env, and add a gitlab personal access token:
TOKEN=glpat-JUST20LETTERSANDNUMB
  • Run git add . and git commit -m 'test' to commit the file.
  • Run the command to push the commited file git push.
  • Verify the push fails due to the secret detected and it shows the correct file path and commit sha.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Ahmed Hemdan

Merge request reports