Skip to content

Draft: Add Push Protection to prevent secret leaks in repositories

Yuanchen Lu requested to merge yuanchenlu/gitlab-ee:master into master

What does this MR do and why?

This MR adds the capability to preemptively detect secret leaks before they are committed to the remote repo using a global server side git hook as an attempt to solve #246819. Github has push protection feature enabled which carries the same idea.

To start, a project setting checkbox is provided to allow opt-in and opt-out. By default, all projects would opt in unless a project admin chooses to opt out.

The secret detection module used here is https://github.com/ibm/detect-secrets with a list of plugins to choose from. We can customize by providing a template and add capability to choose what plugin to use on a per project basis.

I realize in the discussions that server hooks have performance issue so this feature is intended to use for self-hosted gitlab instance only that won't have a high rate of git operations. A feature flag will need to added to enable this feature.

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests. Project settings include a new checkbox defaulting to checked for newly created projects: image Admin area includes a new checkbox to enable/disable secret detection for all projects: image If a secret leak is detected, push will fail with the following message on the CLI (note I don't have the code for rendering UI message yet):

Potential secrets about to be committed to git repo! Please rectify.

Secret Type: JSON Web Token
Location:    token:1

Possible mitigations:

  - Mark false positives with an inline `pragma: allowlist secret`
    comment
  - Commit with `--no-verify` if this is a one-time false positive

If a secret has already been committed, visit
https://help.github.com/articles/removing-sensitive-data-from-a-
repository
Before After

How to set up and validate locally

  1. To set up this feature, a server side hook is required. For self hosted gitlab in k8s, I installed the pre-receive hook to Gitaly's /home/git/custom_hooks/pre-receive.d/.

The following script is what I have:

#!/usr/bin/env bash

PROJECT_ID=`echo ${GL_REPOSITORY} | sed 's/[^0-9]*//g'`
RUN_HOOK=`curl -s -m 2 "<GITLAB_HOSTNAME>/api/v4/projects/${PROJECT_ID}/secret_detection" \
    -H 'PRIVATE-TOKEN: <ADMIN_TOKEN>' | jq -r '.push_protection_enabled'`

# skip secret detection if curl times out
if [ $RUN_HOOK = "1" ]; then
    oldrev=$1
    newrev=$2
    refname=$3
    detect-secrets-hook --fail-on-unaudited `git diff $oldrev $newrev --name-only`
else
    exit 0
fi
  1. create a new project and navigate to its project setting to make sure the checkbox is checked
  2. push a commit containing a secret and observe the operation fail
  3. push a commit not containing a secret and observe the operation pass

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Yuanchen Lu

Merge request reports