## Overview To perform regex matching on git blobs that may include secrets, we plan to create a small Ruby library/gem that will be included as a dependency in GitLab main codebase (`gitlab-org/gitlab`). This dependency will accept one or more git blobs, match them against a defined ruleset of regular expressions (based on `gitleaks.toml` used by [secrets analyzer](https://gitlab.com/gitlab-org/security-products/analyzers/secrets)), and return scan results. ### **Why a gem and not directly in** `gitlab-org/gitlab`'s `/lib` folder? Well, a few things: having this included directly in `gitlab-org/gitlab` defeats the purpose of [continuous secret detection](https://gitlab.com/gitlab-org/gitlab/-/issues/413274), which aims to increase secret detection [coverage](https://about.gitlab.com/direction/secure/static-analysis/secret-detection/#1-year-plan) outside of CI/CD pipelines-context. Additionally, the aim here is to build a self-contained library – similar to [Hush](https://gitlab.com/gitlab-org/secure/pocs/hush) (internal only), or [this POC](https://gitlab.com/gitlab-org/secure/pocs/secret-detection-go-poc) (internal only) extracted from it – that accept code in the form of blobs, and return scan results. Moreover, not using a gem will increase tight coupling and will not make it easy to re-use it elsewhere if the code is only part of `gitlab-org/gitlab` codebase. For more on this point, please make sure to read on the [advantages of using gems](https://docs.gitlab.com/ee/development/gems.html#advantages-of-using-gems), and the pros/cons of having it inside the [same repository](https://docs.gitlab.com/ee/development/gems.html#in-the-same-repo) vs. [external repository](https://docs.gitlab.com/ee/development/gems.html#in-the-external-repo). ### **Shall we use the** `gitleaks.toml` file from `secrets` analyzer? Probably yes, the `secrets` analyzer is likely to continue to be maintained in the long term, and we should also avoid maintaining two separate set of rules (one in the analyzer and one in the gem). Plus, using the same configuration means we achieve some level of parity between our CI-based product offering and this new feature. Finally, our [benchmarking spike](https://gitlab.com/gitlab-org/gitlab/-/issues/423832) based its [results](https://gitlab.com/gitlab-org/gitlab/-/issues/423832#note_1578076768) on [matching blobs against the entire ](https://gitlab.com/gitlab-org/gitlab/-/issues/423832#note_1571210682)`gitleaks.toml` ruleset, therefore, I believe it only make sense to use the same configuration for our implementation. Please read more on this topic from the README of our [Secret Detection Go POC](https://gitlab.com/gitlab-org/secure/pocs/secret-detection-go-poc#ruleset) (internal only). ### **Ruby vs. Go** As explained above, we [decided](https://docs.google.com/document/d/195BM7tAU9iquFkjTROtpuLIlbjfDVKzphZ21ZyzfQlk/edit#bookmark=id.aijtk3i8nlee) (internal only) to use Ruby instead of Go, despite Go having a relatively better performance (check [spike](https://gitlab.com/gitlab-org/gitlab/-/issues/423832) results: [1](https://gitlab.com/gitlab-org/gitlab/-/issues/423832#note_1578076768), [2](https://gitlab.com/gitlab-org/gitlab/-/issues/423832#note_1579850558)). On the other hand, invoking a Go binary from Rails would have required more work to be done, to name a few ([source, internal only](https://docs.google.com/document/d/195BM7tAU9iquFkjTROtpuLIlbjfDVKzphZ21ZyzfQlk/edit#bookmark=id.qv9b66g5mrdx)): * Evaluate execution cost (if running as a daemon). * Figure out communication mode (gRPC vs. shell pipe). * Ensure [security isolation](https://gitlab.com/gitlab-org/gitlab/-/issues/413274#note_1550184207) (internal only). * Hardware resource allocation, and interference with the Rails application resources. Additionally, there was an initial investment involved in making the Go binary official within the GitLab project (e.g. binary distribution, determine release cycles, making it secure, etc). We have also [agreed](https://docs.google.com/document/d/195BM7tAU9iquFkjTROtpuLIlbjfDVKzphZ21ZyzfQlk/edit#bookmark=id.b745m8kfzktj) (internal only) to evaluate the Ruby-based approach, release it gradually behind a feature flag, and monitor it aggressively, while keeping the Go-binary approach on the table in case the former did not have satisfactory results. ### **Ruby's built-in regex matching vs.** `re2` gem While evaluating Ruby vs. Go in [our benchmarking spike](https://gitlab.com/gitlab-org/gitlab/-/issues/423832), @vbhat161 noticed that `gitlab-org/gitlab` project [already uses](https://gitlab.com/gitlab-org/gitlab/-/blob/06f13ecb38c31fcb82c96d98e7ac905f057d76d4/Gemfile#L256) a Ruby wrapper of the popular `re2` C++ library. So, together with @craigmsmith they benchmarked `re2.match` and `RE2::set` from the `re2` gem against Ruby's built-in regex library `Regexp`, and the results where highly in favour of using [precompiled](https://mudge.name/re2/RE2/Set.html#compile-instance_method) `RE2::Set` vs. built-in `Regexp` and `re2.match` method. Therefore, we should use `RE2::Set` and make sure it's precompiled with the regular expressions patterns loaded from `gitleaks.toml` file, before we match them against git blobs inside the gem. Please have a look at the [code they used for benchmarking](https://gitlab.com/craigmsmith/regex-performance) for example, and the results present in [the README](https://gitlab.com/craigmsmith/regex-performance/-/blob/main/README.md?ref_type=heads) for more information. ### **Configurable timeout** We want to make sure we're not blocking pushes in case regex matching is taking too long or there's a failure, therefore, it's important to have a circuit breaker in the gem (configurable via its interface) that stop execution and return early if execution times out. ### **Other considerations** Please check [this section](https://gitlab.com/gitlab-org/gitlab/-/issues/422574#secret-scanner-likely-based-on-httpsgitlabcomgitlab-orggitlab-issues413274) for some other considerations to keep in mind when building the gem.