Draft: Add Push Protection to prevent secret leaks in repositories
What does this MR do and why?
This MR adds the capability to preemptively detect secret leaks before they are committed to the remote repo using a global server side git hook as an attempt to solve #246819 (closed). Github has push protection feature enabled which carries the same idea.
To start, a project setting checkbox is provided to allow opt-in and opt-out. By default, all projects would opt in unless a project admin chooses to opt out.
The secret detection module used here is https://github.com/ibm/detect-secrets with a list of plugins to choose from. We can customize by providing a template and add capability to choose what plugin to use on a per project basis.
I realize in the discussions that server hooks have performance issue so this feature is intended to use for self-hosted gitlab instance only that won't have a high rate of git operations. A feature flag will need to added to enable this feature.
Screenshots or screen recordings
Screenshots are required for UI changes, and strongly recommended for all other merge requests.
Project settings include a new checkbox defaulting to checked for newly created projects:
Admin area includes a new checkbox to enable/disable secret detection for all projects:
If a secret leak is detected, push will fail with the following message on the CLI (note I don't have the code for rendering UI message yet):
Potential secrets about to be committed to git repo! Please rectify.
Secret Type: JSON Web Token
Location: token:1
Possible mitigations:
- Mark false positives with an inline `pragma: allowlist secret`
comment
- Commit with `--no-verify` if this is a one-time false positive
If a secret has already been committed, visit
https://help.github.com/articles/removing-sensitive-data-from-a-
repository
Before | After |
---|---|
How to set up and validate locally
- To set up this feature, a server side hook is required. For self hosted gitlab in k8s, I installed the
pre-receive
hook to Gitaly's/home/git/custom_hooks/pre-receive.d/
.
The following script is what I have:
#!/usr/bin/env bash
PROJECT_ID=`echo ${GL_REPOSITORY} | sed 's/[^0-9]*//g'`
RUN_HOOK=`curl -s -m 2 "<GITLAB_HOSTNAME>/api/v4/projects/${PROJECT_ID}/secret_detection" \
-H 'PRIVATE-TOKEN: <ADMIN_TOKEN>' | jq -r '.push_protection_enabled'`
# skip secret detection if curl times out
if [ $RUN_HOOK = "1" ]; then
oldrev=$1
newrev=$2
refname=$3
detect-secrets-hook --fail-on-unaudited `git diff $oldrev $newrev --name-only`
else
exit 0
fi
- create a new project and navigate to its project setting to make sure the checkbox is checked
- push a commit containing a secret and observe the operation fail
- push a commit not containing a secret and observe the operation pass
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.
Merge request reports
Activity
Hey @yuanchenlu!
Welcome to our community! We're excited to have you here, and can't wait to review this first MR with you!
Thank you for your contribution to GitLab. Please refer to the contribution documentation for an overview of the process.
Did you know about our community forks? Working from there will make your contribution process easier. Please check it out!
When you're ready for a first review, post
@gitlab-bot ready
. If you know a relevant reviewer(s) (for example, someone that was involved in a related issue), you can also assign them directly with@gitlab-bot ready @user1 @user2
.At any time, if you need help, feel free to post
@gitlab-bot help
or initiate a mentor session on Discord. Read more on how to get help.This message was generated automatically. You're welcome to improve it.
added Community contribution workflowin dev labels
assigned to @yuanchenlu
Hey @yuanchenlu!
Welcome to our community! We're excited to have you here, and can't wait to review this first MR with you!
Thank you for your contribution to GitLab. Please refer to the contribution documentation for an overview of the process.
Did you know about our community forks? Working from there will make your contribution process easier. Please check it out!
When you're ready for a first review, post
@gitlab-bot ready
. If you know a relevant reviewer(s) (for example, someone that was involved in a related issue), you can also assign them directly with@gitlab-bot ready @user1 @user2
.At any time, if you need help, feel free to post
@gitlab-bot help
or initiate a mentor session on Discord. Read more on how to get help.This message was generated automatically. You're welcome to improve it.
added 1st contribution label
mentioned in issue gitlab-org/quality/triage-reports#12531 (closed)
added featureaddition typefeature labels
added groupstatic analysis label
added devopssecure sectionsec labels
added linked-issue label
@yuanchenlu, it seems we're waiting on an action from you for approximately two weeks.
- Do you still have capacity to work on this? If not, you might want to close this MR and/or ask someone to take over.
- Do you need help in getting it ready? At any time, you can:
- If you're actually ready for a review, you can post
@gitlab-bot ready
.
This message was generated automatically. You're welcome to improve it.
added automation:author-reminded label
Security policy violations have been resolved.
Edited by GitLab Security Bot@yuanchenlu are you still interested in working on this?
@aharadon Unfortunately I don't have bandwidth to work on this
@yuanchenlu, how was your code review experience with this merge request? Please tell us how we can continue to iterate and improve:
- React with a
or a on this comment to describe your experience. - Create a new comment starting with
@gitlab-bot feedback
below, and leave any additional feedback you have for us in the comment.
Request access to our community forks to receive complimentary access to GitLab Duo, our AI-powered features. With Code Suggestions, Chat, Root Cause Analysis and more AI-powered features, GitLab Duo helps to boost your efficiency and effectiveness by reducing the time required to write and understand code and pipelines. Visit the GitLab Duo documentation to learn more about the benefits.
Subscribe to the GitLab Community Newsletter for contributor-focused content and opportunities to level up.
Thanks for your help!
This message was generated automatically. Improve it or delete it.
- React with a