## Problem to Solve While GitLab provides comprehensive coverage for well-known service providers and standard secret formats, every organization has unique security requirements that demand custom detection rules. Custom rules for secret detection enables organizations to extend GitLab's builtin secret detection capabilities to identify organization specific credential patterns, use and detect custom authentication tokens, and turn off secrets that are generating high false positives. Our customization capabilities span both _Pipeline Secret Detection_ and _Secret Push Protection_, allowing security teams to define custom patterns using regular expressions, manage detection rule, and maintain consistent security policies across their development workflows. This ensures full coverage for all or an organizations secrets. ## Proposal Using configuration profiles, users will be able to add new, regex-based rules that are relevant for their organization. ## User Experience ### Rule Creation Interface - Custom rules are created within the UI and can be added to specific configuration profiles by project. - REST API and GraphQL endpoints are available for rule management, validation and performance testing. ###### Rule Configuration Requirements Users **must** provide all attributes when creating a custom rule: - `title`: Rule name (required) - `description`: Purpose and scope of the rule (required) - `remediation`: Instructions for addressing detected secrets (optional) - `id`: Unique identifier for the rule (required, validated for uniqueness) - `keywords`: Array of keywords to optimize scanning (optional but recommended) - `regex`: Regular expression pattern (required, validated) ###### Rule States - **Active**: Rule is running and scanning code - **Paused**: Rule temporarily disabled due to noise/performance issues - **Draft**: Rule created but not yet validated/enabled - **Disabled**: Rule manually disabled by user ### Rule Validation Process **Frontend Validation (immediate feedback):** - Regex syntax validation - Performance estimation - Pattern optimization suggestions - Duplicate ID detection - Required field validation **Backend Validation (on submission):** - Performance testing against sample datasets - Timeout testing (custom rules have _(To be determined)_ ms timeout limit) ### Error Handling _After rule creation._ ###### Noisy Rules - Rule triggers \>50 false positives per day across organization (or) - User-reported false positive rate \> 80% _User experience when noisy rule detected:_ 1. **Notification**: Email and in-app notification to rule creator and organization admins 2. **Graceful degradation**: Rule automatically paused, not removed 3. **Dashboard status**: Clear indication in UI that rule is paused due to noise 4. **Resolution path**: Users can review detections, refine rule, and try to enable once updates have been made ###### **Performance failures** * **Timeout limit**: _(To be determined)_ ms per custom rule execution * **Memory limit**: _(To be determined)_ MB per rule execution * **Concurrent limit**: Maximum _(To be determined)_ custom rules per organization _User experience when a non-performant rule is detected:_ - **Immediate feedback**: "Custom rule 'API Key Detection Rule' exceeded performance limits and was skipped for this push" - **Push behavior**: Push continues with other rules, custom rule is automatically disabled - **Follow-up**: User receives notification with optimization suggestions - **Recovery**: Rule remains disabled until user updates and validates the rule ## Out of Scope - Rule sharing across organizations, instances and groups - Advanced rule testing environments - Integration with external secret detection tools ### Target Audience 1. [Amy (Application Security Engineer)](https://handbook.gitlab.com/handbook/product/personas/#amy-application-security-engineer) (Primary) wants to implement comprehensive secret detection that covers organization-specific secrets and APIs. Unfortunately default detection rules don't cover internal services, custom authentication systems, or proprietary secret formats. 2. [Priyanka (DevOps Engineer)](https://handbook.gitlab.com/handbook/product/personas/#priyanka-platform-engineer) (Primary) wants to integrate custom secret detection into CI/CD pipelines with minimal developer friction. Balancing security coverage with developer productivity, managing false positives, maintaining detection accuracy is challenging! Ideally Priyanka can implement custom secrets, managing detection rules as code, and automate pattern deployment. 3. [Cameron (Compliance Manager) ](https://handbook.gitlab.com/handbook/product/personas/#cameron-compliance-manager)(Secondary) wants to ensure comprehensive coverage of sensitive data and maintain audit trails. Ideally Cameron can provide compliance coverage for custom secrets, managing detection rules across large organizations. He needs hierarchical pattern management, compliance reporting, audit trail maintenance. ### Pricing and Packaging Custom rules for secret detection is a ~"GitLab Ultimate" feature. ### References - [Customize pipeline secret detection](https://docs.gitlab.com/user/application_security/secret_detection/pipeline/configure/) ~documentation - [Customize rulesets for static analysis](https://docs.gitlab.com/user/application_security/sast/customize_rulesets/) ~documentation - [Custom rulesets for Advanced SAST](https://gitlab.com/gitlab-org/gitlab/-/issues/501128) # Interlock Details #### Executive Summary Users can add new, custom regex rules for secret push protection via configuration profiles. #### Engineering Assessment * We can't provide an accurate engineering assessment until the requirements and UX are finalized. * We're currently only allowing regex patterns that are "well defined" to be used in SPP. If we allow users on `.com` to create their own default rules, this could have a significant performance impact depending on what rules they introduce. This could also cause instability with the SDS or even the Rails platform. We will need to involve the Infrastructure team to understand how we can support this without affecting platform stability. #### Dependencies - Team dependencies: - Epic/Issue dependencies - _Link to dependent epics/issues via the linked items widget below for ease of drill down_ - External dependencies: * ~"group::security platform management" * We'll need a way to load rule customizations during the scan, this will depend on where this data is persisted. * We'll need a way to persist this data and develop a new schema for configuration options. * ~UX https://gitlab.com/gitlab-org/gitlab/-/issues/560643+ * ~infrastructure #### DRIs - **PM**: @abellucci - **EM**: @amarpatel - **UX/PDM**: @mfangman - **Group(s)**: ~"group::secret detection" - **Engineering Owner**: @rvider