## :star: Interested in this feature? :star: Please register interest in the relevant issue/epic: * https://gitlab.com/gitlab-org/gitlab/-/issues/439921+ * https://gitlab.com/groups/gitlab-org/-/epics/13523+ This will help keep the epic easier to read and track. --- ## Overview This epic leads to the next iterative milestone for secret push protection: being available in Beta on GitLab.com. This includes two related but distinct goals: - releasing on .com. - meeting [Beta maturity criteria](https://docs.gitlab.com/ee/policy/experiment-beta-support.html#beta). We decided to work on these goals together because we determined that some of the Beta criteria were necessary before we would feel comfortable expanding pre-GA testing to more customers. That's because this feature intentionally blocks an essential action in the developer workflow. See a description of the intended availability phases in [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/439921 "Customers interested in Secret Push Protection"). ## Scope ### Summary After the work in this epic is complete: 1. GitLab will block commits from being pushed if they contain specific secrets. 1. The specific set of secret patterns will include well-identifiable secrets (those with a defined format) that can be tested efficiently (in milliseconds). 2. This protection will apply no matter how commits are created—via the `git` command line, single-file editor, Web IDE, or other places. 2. Users will be able to bypass this blocking by setting a specific commit message or a push option. This is in accordance with the ["always allow for deploying to production"](https://handbook.gitlab.com/handbook/product/product-principles/#always-allow-for-deploying-to-production) product principle, and is necessary so that teams can confidently enable the feature while minimizing risk to developer productivity. 1. Bypasses will create audit events, and will be reflected in metrics that GitLab can use to update rule quality. 2. Secrets that are successfully pushed (either because of a user bypass, a timeout, or a rule not being usable in pre-receive scanning for performance reasons) will still be detectable through the existing [post-receive Secret Detection scanning](https://docs.gitlab.com/ee/user/application_security/secret_detection/pipeline/index.html), which currently runs in CI/CD pipelines. 3. For system reliability and performance reasons, scanning will be subject to a deadline. If the deadline expires before the scanner has made a decision, the push will be allowed to continue. This is a key reliability control to avoid potential outages. 4. Administrators will be able to enable this protection on projects or groups. 5. The system will have been scale-tested for performance and reliability, and procedures will be in defined for GitLab team members to assure that the system is reliable. ### Details Reaching this goal will require: 1. Visible product changes... 1. ...for users. - Epic: https://gitlab.com/groups/gitlab-org/-/epics/13036+ 2. ...for administrators/those enabling the feature. - Epic: https://gitlab.com/groups/gitlab-org/-/epics/13037+ 2. Behind-the-scenes changes 1. Analysis of system performance and resource requirements. - Epic: https://gitlab.com/groups/gitlab-org/-/epics/12916+ 2. System changes based on that analysis - Epic: https://gitlab.com/groups/gitlab-org/-/epics/12920+ 3. Monitoring and alerting that makes us confident we can enable the feature - Epic: https://gitlab.com/groups/gitlab-org/-/epics/13038+ 4. Event tracking and instrumentation that will help identify and prioritize necessary product changes: - Epic: https://gitlab.com/groups/gitlab-org/-/epics/12917+ ### Useful links - [Group meeting discussion on Beta scope](https://docs.google.com/document/d/1yhO41W5xpdQ5xgCrAydZC1hn8PrqkNWsfBcCL5GJ3F8/edit#bookmark=id.4xds2dti7edt) (internal link) - [Looking forward/refinement meeting](https://docs.google.com/document/d/1nRsp76-TCG3oUTk4245Lw_kvVuj-nc3tSsRCzhE5jXE/edit#bookmark=id.993h5h5ir2xk) (internal link) ### Timeline (Goal Posts) <table> <tr> <th>Target</th> <th>Description</th> <th>Definition of Done</th> <th>DRI</th> <th>Target Delivery Date</th> <th>Status</th> </tr> <tr> <td> Beta is feature complete ([1](https://gitlab.com/groups/gitlab-org/-/epics/13037),[2](https://gitlab.com/groups/gitlab-org/-/epics/13036)) </td> <td>All visible product changes have been implemented, tested, and released.</td> <td> The following sub-epics are delivered and closed: * https://gitlab.com/groups/gitlab-org/-/epics/13036 * https://gitlab.com/groups/gitlab-org/-/epics/13037 </td> <td> @serenafang </td> <td> %"17.0" (2024-05-10) </td> <td> Completed :white_check_mark: </td> </tr> <tr> <td> [Baseline monitoring dashboard and runbook have been created.](https://gitlab.com/groups/gitlab-org/-/epics/13038) </td> <td>Dashboard with system metrics from initial profiling created to use for incidents/issues.</td> <td> 1. Dashboard with system metrics from initial profiling created to use for incidents/issues. 2. Runbook on how to look at and improve the dashboard. </td> <td> @ahmed.hemdan </td> <td> %"17.0" (2024-05-03) </td> <td> Completed :white_check_mark: </td> </tr> <tr> <td> Initial phase of [event tracking](https://gitlab.com/groups/gitlab-org/-/epics/12917) is completed </td> <td> </td> <td> The following metrics or events have been created: * Adoption Metrics 1. Track count of projects with feature enabled. 2. Track count of instances with feature enabled. * Product Metrics 1. [Track count of secrets detected](https://gitlab.com/gitlab-org/gitlab/-/issues/443352 "Track count of detected secrets"). 2. [Track types of secrets blocked and their cardinality](https://gitlab.com/gitlab-org/gitlab/-/issues/443354 "Track types of secrets blocked and their cardinality"). 3. [Track count of suppressed secret detection](https://gitlab.com/gitlab-org/gitlab/-/issues/443353 "Track count of suppressed secret detections"). * System Metrics 1. TBD </td> <td> @eurie </td> <td> %"17.1" </td> <td> **Deferred to GA** - We were able to get a database metric merged in Beta, but ran into a few issues getting this to display in Tableau. We'll continue this effort in GA </td> </tr> <tr> <td> [Dogfooding on internal GitLab projects](https://gitlab.com/groups/gitlab-org/-/epics/13523) </td> <td> We plan to take a phased approach to dogfooding this feature internally to get some early feedback prior to enabling it on `.com` </td> <td> * Phase 1 * Static Analysis/SD analyzers (group). * Expand to other analyzers (stage). * Expand to govern and VR projects (section). * Phase 2 * Collaboration with Appsec * Phase 3 * Key GitLab projects: * [\`gitlab-org/gitlab\`](https://gitlab.com/gitlab-org/gitlab) * [\`gitlab-org/gitaly\`](https://gitlab.com/gitlab-org/gitaly) * [\`gitlab-org/gitlab-runner\`](https://gitlab.com/gitlab-org/gitlab-runner) </td> <td> @rossfuhrman </td> <td> %17.0 - %"17.1" </td> <td> Completed :white_check_mark: </td> </tr> <tr> <td> [Alerting established for key components](https://gitlab.com/gitlab-org/gitlab/-/issues/456770) </td> <td>Since this feature is on a "hot-path", we'd like to establish alerting that would allow us to proactively troubleshoot any performance or stability related issues.</td> <td> * Identify which components to monitor. * Decide on what thresholds to alert on if exceeded. * Decide on what tools will be used (e.g. Sentry vs. Grafana for application-based vs. infrastructure-related monitoring). </td> <td> @ahmed.hemdan </td> <td> %"17.1" (2024-06-14) </td> <td> **Deferred to GA -** We decided to push this to GA in order to focus on more critical deliverables that would support a faster GA delivery. </td> </tr> <tr> <td> [Follow-up issues created from Dogfooding feedback](https://gitlab.com/groups/gitlab-org/-/epics/13884) </td> <td>Review the feedback from our dogfooding phase and create any follow up issues as needed, prioritizing any critical fixes prior to our Beta launch.</td> <td> * Create a feedback issue. * Add follow-up issues to GA epic. * If critical or blocking, work with the PM to prioritize for Beta. </td> <td>Secret Detection team</td> <td> %"17.1" </td> <td> Completed :white_check_mark: </td> </tr> <tr> <td> [Regex pattern improvements](https://gitlab.com/gitlab-org/gitlab/-/issues/435370 "Establish process for new patterns and pattern improvements") </td> <td>We'd like to continue iterating on and improving the patterns we ship with Secret Push Protection. This will allow us to reduce the false positives that are detected early on in the SDLC, as well as improve performance of scans.</td> <td> * Establish criteria for well-defined patterns * Patterns are evaluated based on that criteria * MR containing updated list of patterns is merged * Documentation added for what patterns are being used for the different types of SD scanning. </td> <td> @rossfuhrman </td> <td> %"17.1" </td> <td> **Deferred to GA -** This was a larger effort than expected, and have made it a key deliverable for GA </td> </tr> <tr> <td> **_\[Nice-to-have\]_** [Performance optimizations](https://gitlab.com/groups/gitlab-org/-/epics/13100 "[Post-GA] Scanning performance optimizations") have been refined and prioritized </td> <td>Determine if the suggested performance optimizations could make an impactful improvement on our performance.</td> <td> * Spikes are conducted. * Decide on the prioritization for each based on the spikes. </td> <td> @eurie </td> <td> %"17.1" (2024-06-14) </td> <td> **Descoped** - This will be reprioritized if we encounter any performance concerns through Beta feedback. </td> </tr> <tr> <td> **_\[Nice-to-have\]_** Impactful [performance optimizations ](https://gitlab.com/groups/gitlab-org/-/epics/13100)have been implemented </td> <td>Once identified, implement the applicable performance improvements</td> <td> Follow definition of done defined [here](https://gitlab.com/groups/gitlab-org/-/epics/13100#implementation "[Post-GA] Scanning performance optimizations"). </td> <td>TBD</td> <td> %17.2 (TBD) </td> <td> **Descoped** - This will be prioritized if we encounter any performance concerns through Beta feedback. </td> </tr> <tr> <td> [Beta production readiness](https://gitlab.com/gitlab-org/gitlab/-/issues/451595) has been completed </td> <td>Ensure we meet the requirements required to release this as a Beta feature.</td> <td>Production readiness MR has been reviewed by all relevant groups and merged.</td> <td> @serenafang </td> <td> %"17.1" (2024-06-20) </td> <td> Completed :white_check_mark: </td> </tr> <tr> <td> [Beta launch material prepped](https://gitlab.com/gitlab-org/gitlab/-/issues/454391) </td> <td>Prior to launch, we'll need to put together some materials to ensure the feature is properly documented and announced.</td> <td> * Blog post, marketing strategy, release post, etc. * Decide on [secret detection feature naming](https://gitlab.com/gitlab-org/gitlab/-/issues/454395 "Secret Detection Feature Naming"). * Documentation updates are finalized. * Support readiness issue created/approved. </td> <td> @smeadzinger @amarpatel </td> <td> %"17.1" (2024-06-20) </td> <td> Completed :white_check_mark: </td> </tr> <tr> <td> [Enable feature for `.com` customers](https://gitlab.com/gitlab-org/gitlab/-/work_items/467190) </td> <td> Once we've completed the above tasks, we'll be ready to release this feature for our `.com` customers. </td> <td> * Evaluate whether to do a closed or open beta. * If closed beta, get support from Product on who to enable it for. </td> <td> @djadmin @rossfuhrman </td> <td> %"17.1" </td> <td> Completed :white_check_mark: </td> </tr> </table> _This page may contain information related to upcoming products, features and functionality. It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes. Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc._