## Problem Committing secrets to git repository is an on-going problem, and GitLab has a solution in the form of a secret detection CI job. However this only runs after the code has been added into the repository, potentially exposing it and requiring costly cleanup. You can read more about this challenge here in this blog post: https://about.gitlab.com/blog/2023/01/04/pat-revocation-coming-soon/ Instead of cleaning up, it would be better to enforce secret detection prior to adding changes to the repository. While we will continue to build [our own solution](https://gitlab.com/gitlab-org/gitlab/-/issues/422574) to this problem, we also recognize that a lot of organizations currently rely on separate tools, like [GitGuardian](https://www.gitguardian.com/), to accomplish this. We want to give them a way to leverage their existing tools to accomplish this task. ## Proposed solution We will introduce a new option for self-managed and Dedicated installs to configure an integration with third-party secret scanning tools. The first integration built will be with [GitGuardian](https://www.gitguardian.com/). ### Configuration options - Admin area settings - Add new section for `Security Integrations` - Add option to enable `Pre-receive GitGuardian Secrets Detection` - When enabled, add a required field for users to add an API key - Top-group settings - If `Pre-receive GitGuardian Secrets Detection` is enabled at the admin level, add an option labeled `Pre-receive GitGuardian Secrets Detection` to the group `Settings -> Repository -> Pre-defined push rules` section that is enabled by default. - Add a sub-setting to allow project owners to override this pre-receive scan for their project. - This setting should be disabled by default - Project settings - Add an option for `Pre-receive GitGuardian Secrets Detection` in the project `Settings -> Repository -> Push rules` section - If project owner override is disabled, this setting will be checked and unchangeable. - If project owner override is enabled, this setting will be check and allow project owners to disable it. ### Gitaly implementation Gitaly already has the `/internal/allowed` endpoint which Gitaly contacts prior to accepting changes on a `push` command. This would the the appropriate place to insert a secret scanning mechanism. Additionally, Gitaly already has the means to enumerate new blobs only. This has the added benefit that any such scanning would only scale with the size of the change and not with the size of the repository. Pros: * Checked at the server side, no client configuration / binaries required * Automatically applies to all ways of committing to repositories (clients, Web UI, remote development, etc.) Cons: * While scanning only new blobs should scale well in most cases, there would most likely need to be a way to exclude specific non-performant file types such as binary files, or very large blobs. * Cannot degrade other customers/users experience who may have repositories on the same node * Cannot be a source of alerts and other instability internally * The scanner needs to be trusted code as it will be running on our SaaS services, and will need to meet our code quality and security requirements ### Details #### New built-in push rule We would introduce a new built-in push rule to GitLab, similar to the existing rule which performs checks based on filename: https://docs.gitlab.com/ee/user/project/repository/push_rules.html#prevent-pushing-secrets-to-the-repository #### Usage of Gitaly internal/allowed API to only scan changes in commit This push rule would trigger a new push check using the `/internal/allowed` API: https://docs.gitlab.com/ee/development/internal_api/internal_api_allowed.html#push-checks <details> <summary>Details of the `/internal/allowed` functionality:</summary> First, Gitaly sends the list of new reference tips to the /internal/allowed endpoint, which tells it which references have been updated to what commit. From here on, there are essentially two different ways to enumerate new objects. #### git-rev-list(1)-base enumeration The caller essentially executes git rev-list $new_tips --not --all, which asks Git to enumerate all objects reachable from the new tips while excluding all preexisting objects. This is the obvious solution, but it is unfortunately very slow in large monorepositories. In gitlab-org/gitlab, this enumeration would frequently take around 25-30 seconds. This would be implemented e.g. via ListCommits() or ListBlobs(). #### quarantine-directory-based enumeration Nowadays, all changes in Gitaly are first staged into a quarantine directory that hosts the new objects outside of the main object database. Gitaly sends information about that staging directory to the client. This has two advantages: The client can actually tell Gitaly which objects to enumerate, which is a prerequisite for inspecting the objects in the first place. It is possible to simply enumerate all objects in that quarantine directory, which avoids the graph walk of the first option. This is the recommended method nowadays, where the client will invoke ListAllBlobs() or ListAllCommits() to enumerate all new blobs or commits, respectively. This is way more efficient and scales with the size of the change compared to the number of references. In gitlab-org/gitlab, this has reduced the time to enumerate new objects from 25-30 seconds to about a dozen milliseconds. There are two downsides: The blobs will have no path attached to them. This is not an issue in the context of this issue here though as we want to do content-based scanning anyway. When serving pushes, the client may end up sending more objects than required. This means that some objects might end up being validated multiple times. I don't think this is an issue either though. #### Rails side The Rails side nowadays uses the second approach when available. It still has the first approach in place in case there are calls where there is no quarantine directory available. Traditionally we only did this for listing commits, but in the context of global file size limits we have recently also extended this model to work with blobs. </details> ## Iterations 1. UI and API for Admin area settings 2. UI and API for group settings 3. UI and API for project settings 4. Gitaly `/internal/allowed` push rule