## Summary This issue tracks the rollout of the [`malware` filter argument on the Dependency GraphQL APIs](https://gitlab.com/gitlab-org/gitlab/-/issues/587758) on production, which is currently behind the `malicious_packages_dependency_list_filtering` feature flag. - Feature flag name: `malicious_packages_dependency_list_filtering` - Feature flag type: `wip` - Default enabled: `false` - Introduced in: [!235800](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/235800) - Feature issue: [#587758](https://gitlab.com/gitlab-org/gitlab/-/issues/587758) - Parent epic: [&20573](https://gitlab.com/groups/gitlab-org/-/epics/20573) ## Owners - Team: `group::security infrastructure` - Most appropriate Slack channel to reach out to: `#g_security-infrastructure` - Best individual to reach out to: @bala.kumar ## Expectations ### What are we expecting to happen? - When the flag is **disabled** (default): the `malware` argument on `Query.project.dependencies` and `Query.group.dependencies` raises `Gitlab::Graphql::Errors::ArgumentError` with the message `"The malware filter is not available."`. - When the flag is **enabled**: the argument is accepted by the resolver. Until the finder integration follow-up lands, the argument is a no-op and dependencies are returned unfiltered. - Once the finder integration is implemented in a follow-up MR, enabling the flag will activate actual filtering of dependencies by malware status (CWE-506 / `GLAM-*` identifier prefix), gated by the SSCS add-on license check. ### What can go wrong and how would we detect it? - The GraphQL schema introspection picks up the new experimental argument: detectable via existing GraphQL schema snapshot specs. - Once filtering is wired in, the PG join on identifiers may hit performance limits ([&17619](https://gitlab.com/groups/gitlab-org/-/work_items/17619)). Monitor: - SQL slow log entries on `sbom_occurrences` joins. - Apdex / latency on the dependency list endpoints in [Grafana](https://dashboards.gitlab.net/). - Sidekiq / web error rates on `Resolvers::Sbom::DependenciesResolver` and `Resolvers::Sbom::DependencyAggregationResolver`. - Rollback by disabling the flag immediately restores prior behaviour (no schema breakage since the argument is `experiment`). ## Interaction with dependency_malware_detection The dependency list malware filter token in the UI is gated by a different flag, `dependency_malware_detection` (rollout issue #611434), not by this one. If `dependency_malware_detection` is enabled while this flag is not, the token is still rendered, and applying it raises the GraphQL error `The malware filter is not available.`, raised by `validate_malware_filter!`. These two flags should be enabled together, with this one enabled first. Note that even with both enabled, filtering is a no-op until the finder integration in #587758 lands. ## Rollout Steps Note: Please make sure to run the chatops commands in the Slack channel that gets impacted by the command. ### Rollout on non-production environments - Verify the MR with the feature flag is merged to `master` and has been deployed to non-production environments with `/chatops gitlab run auto_deploy status <merge-commit-of-your-feature>` - [ ] Deploy the feature flag at a percentage (recommended percentage: 50%) with `/chatops gitlab run feature set malicious_packages_dependency_list_filtering 50 --actors --dev --pre --staging --staging-ref` - [ ] Monitor that the error rates did not increase (repeat with a different percentage as necessary). - [ ] Enable the feature globally on non-production environments with `/chatops gitlab run feature set malicious_packages_dependency_list_filtering true --dev --pre --staging --staging-ref` - [ ] Verify that the feature works as expected on `staging-canary`. ### Before production rollout - [ ] Ensure the finder integration follow-up has been merged and deployed before enabling on production groups. - [ ] Coordinate with @dpisek and the frontend ([#587762](https://gitlab.com/gitlab-org/gitlab/-/issues/587762)) team before flipping on customer-facing groups. ### Specific rollout on production For visibility, all `/chatops` commands that target production must be executed in the [`#production` Slack channel](https://gitlab.slack.com/archives/C101F3796) and cross-posted (with the command results) to the responsible team's Slack channel. - [ ] Enable for `gitlab-org` and `gitlab-com` first: - `/chatops gitlab run feature set --group=gitlab-org,gitlab-com malicious_packages_dependency_list_filtering true` - [ ] Verify that the feature works for the specific actors via GraphiQL on `Query.project.dependencies(malware: true)` and `Query.group.dependencies(malware: true)`. ### Preparation before global rollout - [ ] Set a milestone to this rollout issue to signal for enabling and removing the feature flag when it is stable. - [ ] Ensure documentation is in place for the `malware` argument once filtering is wired up. - [ ] Notify the [`#support_gitlab-com` Slack channel](https://gitlab.slack.com/archives/C4XFU81LG) and `#g_security-infrastructure`. ### Global rollout on production - [ ] [Incrementally roll out](https://docs.gitlab.com/development/feature_flags/controls/#process) the feature on production: - `/chatops gitlab run feature set malicious_packages_dependency_list_filtering 25 --actors` - `/chatops gitlab run feature set malicious_packages_dependency_list_filtering 50 --actors` - `/chatops gitlab run feature set malicious_packages_dependency_list_filtering 100 --actors` - Between every step wait for at least 15 minutes and monitor the appropriate graphs on https://dashboards.gitlab.net. - [ ] After the feature has been 100% enabled, wait for at least one day before releasing the feature. ### Release the feature - [ ] Create a merge request to remove the `malicious_packages_dependency_list_filtering` feature flag. The MR should: - Remove all references to the feature flag from the codebase (including `validate_malware_filter!` in `Resolvers::Sbom::DependencyInterfaceResolver`). - Remove the YAML definition `config/feature_flags/wip/malicious_packages_dependency_list_filtering.yml`. - Remove the `experiment: { milestone: '19.0' }` marker from the `malware` argument once stable. - [ ] Close [the feature issue](https://gitlab.com/gitlab-org/gitlab/-/issues/587758). - [ ] Once the cleanup MR has been deployed to production, clean up the feature flag from all environments: `/chatops gitlab run feature delete malicious_packages_dependency_list_filtering --dev --pre --staging --staging-ref --production` - [ ] Close this rollout issue. ## Rollback Steps - [ ] This feature can be disabled on production by running: ``` /chatops gitlab run feature set malicious_packages_dependency_list_filtering false ``` - [ ] Disable the feature flag on non-production environments: ``` /chatops gitlab run feature set malicious_packages_dependency_list_filtering false --dev --pre --staging --staging-ref ``` - [ ] Delete feature flag from all environments: ``` /chatops gitlab run feature delete malicious_packages_dependency_list_filtering --dev --pre --staging --staging-ref --production ```