<details> <summary>Table of Contents</summary> [[_TOC_]] </details> # :lock: Interlock details --- ## Executive Summary Secret Detection pipeline scanning is a critical security capability that identifies leaked secrets (API keys, passwords, tokens) in repository history and pipeline artifacts. However, its current enablement experience requires manual CI/CD configuration, creating barriers for security teams trying to deploy comprehensive secret detection coverage at scale. While Secret Push Protection prevents secrets from being committed in real-time, **Secret Detection pipeline scanning** provides complementary protection by: - Scanning the entire repository history for previously committed secrets - Detecting secrets in pipeline artifacts and logs - Identifying secrets that may have bypassed push protection - Providing historical context and remediation guidance This epic delivers the **Secret Detection pipeline scanning implementation** of the basic scan configuration profiles framework, working alongside Secret Push Protection to provide defense-in-depth for secret management. # :chart_with_upwards_trend: Target Metrics 1. **Overall Target: 70% of Ultimate tier active projects adopting 2+ security scanners (**[**Tableau**](https://10az.online.tableau.com/#/site/gitlab/views/High-LevelSecurityMetricCharts/ofGitLab_comUltimateProjectswith2SecurityScannersRunning?:iid=1)**)** 1. Incremental target: Increase % of active Ultimate projects with SD Pipeline scanning enabled to 60% within 3 months following release) ([{-Tableau-}](https://10az.online.tableau.com/#/site/gitlab/views/PDSecSecureMetrics/SecretPushProtectionandSecretDetectionPipeline?:iid=2)) ## Dependencies - Team dependencies: - Security Inventory - Project-level `secret_push_protection_enabled` setting - Security Manager role and permissions - Group-level Security Configuration page - Epic/Issue dependencies - _Link to dependent epics/issues via the linked items widget below for ease of drill down_ - External dependencies: - ~&quot;group::secret detection&quot; - DRI for Q1 supporting in guidance, defining default configurations, planning for custom configurations, aligning on architectural schema ## Engineering Assessment See [Scan Profiles Vision](https://docs.google.com/presentation/d/1dR2z-809QPPx1bkVks5F2MY5gRb-YhOeESKUONW4wVU/edit?slide=id.g2df33939f65_0_1787#slide=id.g2df33939f65_0_1787), which includes full project delivery plan and dependencies. ## DRIs - **PM**: @m-omokoh - **EM**: @or.gal @minac - **UX/PDM**: @mfangman - **Group(s)**: ~&quot;group::security platform management&quot; - **Engineering Owner**: @rvider ## Initiative Driver - Product or Engineering? - [x] **Product-driven initiatives (P1/P2/P3)** - Customer-facing features or improvements driven by Product teams that require engineering resources and commitment - These initiatives require a Product Priority label (P1/P2/P3) - They may also receive GTM tier labels (T1/T2/T3) for external communication - [ ] **Engineering-driven initiatives (E1/E2/E3)** - Internal technical improvements that may not have customer-facing components - These initiatives require an Engineering Priority label (E1/E2/E3) - They have internal visibility only and are not externally communicated - Examples include: technical debt reduction, infrastructure improvements, refactoring, dependency upgrades --- # :package: Product Delivery Plans # Acceptance Criteria 1. Security managers can enable Secret Detection pipeline scanning for multiple projects simultaneously through the Security Inventory interface 2. Projects receive Secret Detection pipeline scanning without developers needing to modify `.gitlab-ci.yml` files 3. Development teams can disable Secret Detection pipeline scanning profile at the project level if they experience negative impacts 4. Secret Detection runs automatically on pipelines and scans repository history 5. Uses identical interaction patterns as Secret Push Protection and other scan profiles 6. Clear differentiation in UI between Secret Push Protection and Secret Detection pipeline scanning # Intended users - [Amy, Application Security Engineer](/handbook/product/personas/#amy-application-security-engineer) - Primary: Needs comprehensive secret detection coverage across all projects - [Alex, Security Operations Engineer](/handbook/product/personas/#alex-security-operations-engineer) - Secondary: Monitors for leaked secrets and compliance - [Sasha, Software Developer](/handbook/product/personas/#sasha-software-developer) - Affected: Receives secret detection findings in merge requests # User experience goal **For Security Managers:** 1. Navigate to Security Inventory at the group level 2. Filter/select target projects (e.g., all projects without Secret Detection pipeline scanning) 3. Bulk apply "Secret Detection Pipeline Scanning - Default" profile 4. View coverage metrics showing which projects have both Secret Push Protection AND Secret Detection pipeline scanning enabled 5. Understand the complementary nature of both secret detection methods **For Developers:** 1. Secret Detection runs automatically on merge request pipelines 2. Findings appear in merge request security widget (clearly labeled as "Pipeline Scanning") 3. Can view applied profile in project Security Configuration page 4. Option to remove profile if experiencing issues (e.g., false positives, pipeline performance) 5. Clear guidance on difference between push protection blocks vs. pipeline scan findings # Detailed Proposal ## Profile Structure - **Profile Name**: "Secret Detection Pipeline Scanning - Default" - **Configuration**: - Uses latest Secret Detection template (`Security/Secret-Detection.gitlab-ci.yml`) - Scans repository history and pipeline artifacts - Default ruleset (no customization in v1) - Runs on merge request pipelines and default branch - **Versioning**: Tied to `latest` scanner version - **Scope**: Can be applied at group or project level ## Key Differentiators from Secret Push Protection | Feature | Secret Push Protection | Secret Detection Pipeline Scanning | |---------|------------------------|------------------------------------| | **When it runs** | Pre-commit (blocks push) | During CI/CD pipeline | | **What it scans** | Uncommitted changes | Repository history + pipeline artifacts | | **User impact** | Blocks commits immediately | Reports findings in MR widget | | **Coverage** | New commits only | Historical + new commits | | **Enablement** | Project setting toggle | CI/CD pipeline job | ## Implementation Approach ### Backend Mechanism 1. **Policy Engine Integration**: When profile is applied, instantiate a backend security policy 2. **CI Template Injection**: Policy injects Secret Detection template into project pipelines 3. **No Policy Project Required**: Policy configuration stored in centralized database 4. **Trigger Strategy**: - One-time full repository scan on default branch upon enablement - Automatic execution on subsequent merge request pipelines - Batched scheduling across projects to avoid infrastructure overload ### Scanning Scope - **Repository History**: Scans all commits in repository (configurable depth) - **Pipeline Artifacts**: Scans build artifacts for exposed secrets - **Merge Request Changes**: Scans new commits in MR - **Default Branch**: Regular scans on default branch to catch new secrets ## Group-level Management ### Security Configuration Page - Display "Secret Detection Pipeline Scanning - Default" profile in read-only mode - Show profile details: - What it scans (repository history, artifacts, MR changes) - How it differs from Secret Push Protection - Default configuration settings - Projects currently using this profile - **Metrics Dashboard**: - Total projects with Secret Detection pipeline scanning enabled - % coverage across group - Comparison with Secret Push Protection coverage - Secret findings trends over time ### Security Inventory Integration - **New Column**: "Secret Detection Pipeline" status (Enabled/Disabled) - **Combined View**: Show both "Secret Push Protection" and "Secret Detection Pipeline" columns side-by-side - **Bulk Actions**: - Select multiple projects - "Apply Secret Detection Pipeline Profile" action - Preview: Show estimated scan time and resource impact - Confirmation dialog with pipeline impact details - **Filtering**: - Filter by Secret Detection pipeline status - Filter for projects with only push protection (missing pipeline scanning) - Filter for projects with neither method enabled - **Hover States**: - Aggregated: "X% of selected projects have Secret Detection pipeline scanning enabled" - Single project: "Enabled via profile | Last scan: 2 hours ago | 3 secrets found" ## Project-level Management ### Security Configuration Page - **Profile Display**: - "Secret Detection Pipeline Scanning - Default" profile shown in "Scan Profiles Applied" section - Shows: Applied by, Applied date, Last scan time, Findings count - Link to latest Secret Detection findings - Clear distinction from Secret Push Protection setting - **Remove Profile**: - Maintainer+ can remove profile - Warning: "Removing this profile will disable Secret Detection pipeline scanning. Secret Push Protection will remain active." - Option to provide feedback on removal reason ### Pipeline Integration - Secret Detection job appears in pipeline view with `[profile]` indicator - Jobs run in `test` stage by default - Findings appear in merge request security widget with "Pipeline Scanning" label - Historical scan results available in Security Dashboard - No manual `.gitlab-ci.yml` changes required ## Permissions - **Apply Profile (Group-level)**: Security Manager role required - **Apply Profile (Project-level)**: Maintainer or Owner role required - **Remove Profile (Project-level)**: Maintainer or Owner role required - **View Findings**: Developer+ can view Secret Detection results # Technical Considerations ## Scalability - **Batched Enablement**: Apply to max 100 projects per batch - **Staggered Initial Scans**: Schedule full repository scans over 48-72 hours to avoid infrastructure overload - **Incremental Scanning**: After initial scan, only scan new commits - **Resource Limits**: Monitor CI/CD minute consumption and provide warnings ## Performance Impact - **Initial Full Scan**: May take 5-30 minutes depending on repository size - **Incremental Scans**: Typically 1-3 minutes for MR pipelines - **Optimization**: Use shallow clones and caching where possible - **Parallel Execution**: Run Secret Detection in parallel with other security scans ## Conflict Resolution - **Existing Secret Detection Jobs**: Profile is additive - won't conflict with existing configurations - **Duplicate Detection**: UI shows if Secret Detection is already enabled via other methods - **Migration Path**: Users can migrate from manual Secret Detection to profile-based ## Integration with Secret Push Protection - **Complementary Coverage**: Both methods work together for defense-in-depth - **Unified Reporting**: Security Dashboard shows findings from both methods - **Coordinated Enablement**: Recommend enabling both for comprehensive protection - **Clear Messaging**: UI clearly distinguishes between the two methods # User Flows ## Flow 1: Security Manager Enables Secret Detection Pipeline Scanning at Scale 1. Amy navigates to Group \> Security \> Inventory 2. Filters for projects that have Secret Push Protection but not Secret Detection pipeline scanning 3. Selects 100 projects 4. Clicks "Apply Profile" → Selects "Secret Detection Pipeline Scanning - Default" 5. Preview shows: "Initial full scans will be scheduled over 72 hours. Estimated total scan time: 50 hours across all projects." 6. Confirms action 7. Background job schedules initial scans (batched) 8. Amy receives notification when enablement complete 9. Dashboard shows Secret Detection coverage increased from 30% to 95% ## Flow 2: Developer Experiences Secret Detection in MR 1. Sasha creates merge request with code changes 2. Pipeline runs automatically with Secret Detection job (injected by profile) 3. Secret Detection finds AWS access key in commit history 4. Finding appears in MR security widget labeled "Secret Detection (Pipeline Scanning)" 5. Sasha reviews finding, rotates the key, removes from history 6. Re-runs pipeline, secret no longer detected 7. MR approved and merged ## Flow 3: Developer Removes Profile Due to False Positives 1. Sasha's team experiences high false positive rate (test fixtures flagged as secrets) 2. Team wants to configure custom rules 3. Sasha navigates to Project \> Security \> Configuration 4. Sees "Secret Detection Pipeline Scanning - Default" profile applied by Amy 5. Clicks "Remove Profile" 6. Selects reason: "Need custom configuration for false positives" 7. Confirms removal 8. Secret Detection jobs no longer run in pipelines 9. Amy receives notification with removal reason 10. Team configures custom Secret Detection in `.gitlab-ci.yml` with exclusions ## Flow 4: Viewing Combined Secret Detection Coverage 1. Amy navigates to Group \> Security \> Inventory 2. Views combined coverage metrics: - Secret Push Protection: 85% of projects - Secret Detection Pipeline: 75% of projects - Both methods: 70% of projects - Neither method: 10% of projects 3. Filters for projects with neither method enabled 4. Bulk applies both profiles to achieve comprehensive coverage # Migration Paths ## From Manual Secret Detection Configuration 1. **Detection**: UI detects existing Secret Detection configuration in `.gitlab-ci.yml` 2. **Notification**: "This project already has Secret Detection configured manually" 3. **Options**: - Keep manual configuration (don't apply profile) - Apply profile (runs both - additive) - Migrate to profile (remove manual config, apply profile) ## To Advanced Configuration Users needing customization can: 1. Remove profile 2. Use Security Policies for advanced Secret Detection configuration with custom rules 3. Manually configure Secret Detection in `.gitlab-ci.yml` with exclusions and custom patterns ## Quantitative Metrics 1. **Adoption**: 35% of Ultimate customers with 10+ projects have Secret Detection pipeline profile applied within 3 months 2. **Combined Coverage**: 60% of projects have both Secret Push Protection AND Secret Detection pipeline scanning enabled 3. **Secret Discovery**: 40% increase in historical secrets detected across customer base 4. **Remediation**: 50% of detected secrets remediated within 7 days 5. **Retention**: \<12% of applied profiles are removed within 30 days ## Qualitative Metrics 1. **User Understanding**: Clear differentiation between push protection and pipeline scanning 2. **Developer Experience**: Positive feedback on automatic enablement and finding quality 3. **Performance Impact**: \<15% of removals due to pipeline performance concerns # Feature Usage Metrics 1. `profiles_secret_detection_pipeline_applied_count` - Total profiles applied 2. `profiles_secret_detection_pipeline_removed_count` - Total profiles removed 3. `profiles_secret_detection_pipeline_scans_executed` - Total scans triggered by profiles 4. `profiles_secret_detection_pipeline_secrets_found` - Secrets detected via profile-enabled scanning 5. `profiles_secret_detection_combined_coverage` - % of projects with both methods enabled 6. `profiles_secret_detection_pipeline_vs_push_protection` - Comparison metrics 7. `profiles_secret_detection_historical_secrets_found` - Secrets found in repository history # Documentation Requirements 1. **Concept Documentation**: - What is Secret Detection pipeline scanning - How it differs from Secret Push Protection - When to use each method (or both) 2. **How-to Guides**: - Enable Secret Detection pipeline scanning at scale using profiles - Understand Secret Detection findings in merge requests - Remove Secret Detection pipeline profile from a project - Migrate from manual Secret Detection to profile-based - Configure both Secret Push Protection and Secret Detection pipeline scanning 3. **Reference**: - Default Secret Detection configuration - Supported secret types and patterns - Performance impact and optimization tips 4. **Troubleshooting**: - Secret Detection job failures - High false positive rates - Pipeline performance issues - Handling historical secrets # Dependencies - Secret Push Protection profile (establishes patterns and provides complementary coverage) - Security Inventory - Security Manager role - Policy engine for CI injection - Secret Detection templates and scanner # Risks and Mitigations | Risk | Impact | Mitigation | |------|--------|------------| | Pipeline performance degradation | High | Staggered rollout, incremental scanning, easy removal | | High false positive rates | Medium | Use latest scanner, clear documentation on tuning | | CI/CD minute consumption concerns | High | Provide usage estimates, warnings before bulk application | | Confusion with Secret Push Protection | Medium | Clear UI differentiation, comprehensive documentation | | Infrastructure overload from full scans | High | Batched scheduling, staggered initial scans | | Historical secrets overwhelming teams | Medium | Prioritization guidance, remediation workflows | # Out of Scope (Future Iterations) 1. Custom secret patterns and rules 2. Exclusion paths and false positive tuning 3. Scan depth configuration (full history vs. shallow) 4. Integration with secret management tools 5. Automated secret rotation workflows 6. Historical trend analysis and reporting # Related Epics - Parent: [Basic enablement-only scan profiles](/groups/gitlab-org/-/epics/19572) - Sibling: [Enablement-only secret push protection profile](/groups/gitlab-org/-/epics/19802) - Sibling: [Enablement-only Dependency Scanning profile](/groups/gitlab-org/-/epics/19952) - Following: \[Enablement-only SAST profile\] - Following: \[Enablement-only Container Scanning profile\] --- > [!important] > > This page may contain information related to upcoming products, features and functionality. It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes. Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc. <!--triage-serverless v3 PLEASE DO NOT REMOVE THIS SECTION--> > [!important] > > This page may contain information related to upcoming products, features and functionality. It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes. Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc. <!--triage-serverless v3 PLEASE DO NOT REMOVE THIS SECTION-->