## Summary Roll out [the feature](https://gitlab.com/gitlab-org/gitlab/-/work_items/623538) currently behind the `reindex_malware_advisory_occurrence_refs` feature flag. - DRI: @srushik - Team Slack channel: `#sec-security-foundations` The flag gates publishing `PackageMetadata::MalwareAdvisoryStatusChangedEvent` during malware advisory ingestion. A subscriber re-indexes the affected `Sbom::OccurrenceRef` documents so the Elasticsearch `malware` field follows advisory changes instead of waiting for the project's next pipeline. > [!note] > Process and guidance live in the docs — this issue is just the commands and a place to track the rollout. > "Rolling out" means incrementally enabling the flag on GitLab.com to validate stability — it is not the same as releasing the feature, which happens when the flag is removed. > [Feature flag controls](https://docs.gitlab.com/development/feature_flags/controls/) · [Feature flag lifecycle](https://handbook.gitlab.com/handbook/product-development/how-we-work/product-development-flow/feature-flag-lifecycle/#feature-flag-lifecycle) ## What could go wrong? **This flag is boolean only — do not attempt a percentage rollout.** It is checked with the `:instance` actor, matching `PackageMetadata::GlobalMalwareAdvisoryScanWorker`. A `percentage_of_time` rollout would re-roll per call, and there is no re-scan path: an advisory whose roll loses stays stale in the index until its project pushes a new SBOM. **Enable only after a bootstrap advisory sync has completed.** A first/full sync inserts roughly 365k advisories, all of which count as newly flagging, so the flag being on during one would publish an event per advisory. Steady-state delta syncs carry only changed advisories, and most malware packages are typosquats that match no component, so the typical job is one indexed lookup that misses and exits. Check the malware advisory checkpoint is past its first sync before enabling. **Blast radius is bounded on both sides.** The subscription is gated on `Search::Elastic::SbomOccurrenceRefIndexHelper.indexing_allowed?`, so nothing is enqueued where the index does not exist, and `PackageMetadata::ReindexMalwareAdvisoryOccurrenceRefsWorker` runs at `urgency :low` with `concurrency_limit 10`. It only writes to the Elasticsearch bookkeeping queue — no database writes, so there is no data-loss risk and a rollback is simply turning the flag off. Watch Sidekiq queue depth for `package_metadata_reindex_malware_advisory_occurrence_refs` and the Elasticsearch bookkeeping backlog on https://dashboards.gitlab.net. ## Events - [Exceptions with reindex_malware_advisory_occurrence_refs:1](https://log.gprd.gitlab.net/app/discover#/?_g=(time:(from:now-1d,to:now))&_a=(index:'7092c4e2-4eb5-46f2-8305-a7da2edad090',query:(language:kuery,query:'json.exception.feature_flag_states.keyword:%22reindex_malware_advisory_occurrence_refs:1%22'),columns:!(json.exception.class,json.exception.message,json.extra.workflow_id,json.meta.caller_id),sort:!(!(json.time,desc)))) - [Events with reindex_malware_advisory_occurrence_refs:1](https://log.gprd.gitlab.net/app/discover#/?_g=(time:(from:now-1d,to:now))&_a=(index:'7092c4e2-4eb5-46f2-8305-a7da2edad090',query:(language:kuery,query:'json.feature_flag_states:%22reindex_malware_advisory_occurrence_refs:1%22'),columns:!(json.meta.caller_id,json.feature_flag_states),sort:!(!(json.time,desc)))) - [Error rate and other graphs by modifying the examples in the Visualization Library](https://log.gprd.gitlab.net/app/visualize#/?_g=h@358d019&s=FF%20Observability) Feature Flag events are only logged by default for feature flags marked for the current or future milestones. To enable while the feature flag is active, see https://docs.gitlab.com/development/feature_flags/#logging ## Rollout Run all production `/chatops` in [`#production`](https://gitlab.slack.com/archives/C101F3796) and cross-post the results to `#sec-security-foundations`. Background: [incremental rollout process](https://docs.gitlab.com/development/feature_flags/controls/#process). **Non-production** ``` /chatops gitlab run feature set reindex_malware_advisory_occurrence_refs true --dev --pre --staging --staging-ref ``` **Production** — boolean only, see "What could go wrong?" above: ``` /chatops gitlab run feature set reindex_malware_advisory_occurrence_refs true ``` ## Before global rollout Confirm the relevant gotchas before going to 100% — see [enabling a feature for GitLab.com](https://docs.gitlab.com/development/feature_flags/controls/#enabling-a-feature-for-gitlabcom): - [ ] A bootstrap malware advisory sync has completed, so enabling will not publish an event per catalogue entry - [ ] Sidekiq queue depth and Elasticsearch bookkeeping backlog are healthy on staging after the flag is on - [ ] [Docs + version history](https://docs.gitlab.com/development/documentation/feature_flags/) updated, if applicable ## Cleanup Remove the flag once [deemed stable](https://handbook.gitlab.com/handbook/product-development/how-we-work/product-development-flow/feature-flag-lifecycle/#feature-flag-lifecycle) — see [cleaning up](https://docs.gitlab.com/development/feature_flags/controls/#cleaning-up). Remove the flag and its YAML definition from the codebase, then: ``` /chatops gitlab run release check <merge-request-url> <milestone> /chatops gitlab run feature delete reindex_malware_advisory_occurrence_refs --dev --pre --staging --staging-ref --production ``` ## Rollback Turning the flag off stops new events being published. Documents already re-indexed stay correct; documents that would have been corrected go back to being stale until their project pushes a new SBOM. ``` /chatops gitlab run feature set reindex_malware_advisory_occurrence_refs false # production /chatops gitlab run feature set reindex_malware_advisory_occurrence_refs false --dev --pre --staging --staging-ref # non-production /chatops gitlab run feature delete reindex_malware_advisory_occurrence_refs --dev --pre --staging --staging-ref --production # remove entirely ```