## Summary Roll out [the feature](https://gitlab.com/gitlab-org/gitlab/-/work_items/628409) currently behind the `lean_mr_security_report_comparison` feature flag. `Vulnerabilities::CompareSecurityReportsService` builds the added/fixed findings shown on an MR's security tab; today it loads every finding from the base and head pipelines with all columns and preloads, including the large `finding_data` blob, even though only uuid, severity, and scanner id are needed to compute the diff. With the flag on, the finder selects only the columns needed for diffing and skips preloads, then reloads full associations for just the capped added/fixed lists (25 findings each, or 1000 with `security_mr_reports_tab_full_data_set`) before serialization. - DRI: @hptrs - Team Slack channel: `#g_vulnerability_management` > [!note] > Process and guidance live in the docs — this issue is just the commands and a place to track the rollout. > "Rolling out" means incrementally enabling the flag on GitLab.com to validate stability — it is not the same as releasing the feature, which happens when the flag is removed. > [Feature flag controls](https://docs.gitlab.com/development/feature_flags/controls/) · [Feature flag lifecycle](https://handbook.gitlab.com/handbook/product-development/how-we-work/product-development-flow/feature-flag-lifecycle/#feature-flag-lifecycle) ## What could go wrong? This is a read path; it does not write anything, so the blast radius is limited to what the MR security tab shows. The worst realistic outcome is a wrong or empty added/fixed list — missing findings, findings shown that shouldn't be, or a serialization error if a lean finding reaches the serializer without being reloaded. Two smaller behavior changes are gated by the same flag: partial-scan filtering matches on `scanner_id` instead of the scanner's external id, and the "requires manual resolution" check reads the comparison's report type instead of each finding's — either could shift edge-case results. Watch Sidekiq and Rails error rates on https://dashboards.gitlab.net and the `vulnerability_report_branch_comparison` metric emitted by the comparer. Note that results are cached via `ReactiveCaching` in `Security::MergeRequestSecurityReportGenerationService`, so a bad comparison can keep showing up on affected MRs for the cache lifetime even after the flag is turned off. ## Events - [Exceptions with lean_mr_security_report_comparison:1](https://log.gprd.gitlab.net/app/discover#/?_g=(time:(from:now-1d,to:now))&_a=(index:'7092c4e2-4eb5-46f2-8305-a7da2edad090',query:(language:kuery,query:'json.exception.feature_flag_states.keyword:%22lean_mr_security_report_comparison:1%22'),columns:!(json.exception.class,json.exception.message,json.extra.workflow_id,json.meta.caller_id),sort:!(!(json.time,desc)))) - [Events with lean_mr_security_report_comparison:1](https://log.gprd.gitlab.net/app/discover#/?_g=(time:(from:now-1d,to:now))&_a=(index:'7092c4e2-4eb5-46f2-8305-a7da2edad090',query:(language:kuery,query:'json.feature_flag_states:%22lean_mr_security_report_comparison:1%22'),columns:!(json.meta.caller_id,json.feature_flag_states),sort:!(!(json.time,desc)))) - [Error rate and other graphs by modifying the examples in the Visualization Library](https://log.gprd.gitlab.net/app/visualize#/?_g=h@358d019&s=FF%20Observability) Feature Flag events are only logged by default for feature flags marked for the current or future milestones. To enable while the feature flag is active, see https://docs.gitlab.com/development/feature_flags/#logging ## Rollout Run all production `/chatops` in [`#production`](https://gitlab.slack.com/archives/C101F3796) and cross-post the results to `#g_vulnerability_management`. Background: [incremental rollout process](https://docs.gitlab.com/development/feature_flags/controls/#process), [feature actors](https://docs.gitlab.com/development/feature_flags/#feature-actors). **Non-production** ``` /chatops gitlab run feature set lean_mr_security_report_comparison 50 --actors --dev --pre --staging --staging-ref /chatops gitlab run feature set lean_mr_security_report_comparison true --dev --pre --staging --staging-ref ``` **Production** — percentage rollout (wait ≥15 min between steps, watch dashboards): ``` /chatops gitlab run feature set lean_mr_security_report_comparison <percentage> --actors ``` Or target specific actors instead: ``` /chatops gitlab run feature set --project=gitlab-org/gitlab,gitlab-org/gitlab-foss lean_mr_security_report_comparison true /chatops gitlab run feature set --group=gitlab-org,gitlab-com lean_mr_security_report_comparison true /chatops gitlab run feature set --user=hptrs lean_mr_security_report_comparison true ``` ## Before global rollout Confirm the relevant gotchas before going to 100% — see [enabling a feature for GitLab.com](https://docs.gitlab.com/development/feature_flags/controls/#enabling-a-feature-for-gitlabcom): - [Docs + version history](https://docs.gitlab.com/development/documentation/feature_flags/) updated - [Breaking changes](https://docs.gitlab.com/development/documentation/release_notes/#deprecations-removals-and-breaking-changes) announced, if any - [Change management issue](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/#feature-flags-and-the-change-management-process) opened, if required - [External API consumers](https://docs.gitlab.com/development/feature_flags/#do-not-use-feature-flags-in-external-api-consumers) handled with a fail-open mechanism, if applicable ## Cleanup Remove the flag once [deemed stable](https://handbook.gitlab.com/handbook/product-development/how-we-work/product-development-flow/feature-flag-lifecycle/#feature-flag-lifecycle) — see [cleaning up](https://docs.gitlab.com/development/feature_flags/controls/#cleaning-up). Track it here, or open a follow-up [Feature Flag Cleanup issue](https://gitlab.com/gitlab-org/gitlab/-/work_items/new?description_template=Feature%20Flag%20Cleanup). Remove the flag and its YAML definition from the codebase, then: ``` /chatops gitlab run release check <merge-request-url> <milestone> /chatops gitlab run feature delete lean_mr_security_report_comparison --dev --pre --staging --staging-ref --production ``` ## Rollback ``` /chatops gitlab run feature set lean_mr_security_report_comparison false # production /chatops gitlab run feature set lean_mr_security_report_comparison false --dev --pre --staging --staging-ref # non-production /chatops gitlab run feature delete lean_mr_security_report_comparison --dev --pre --staging --staging-ref --production # remove entirely ```