## Summary This issue is to roll out [the feature](https://gitlab.com/gitlab-org/gitlab/-/work_items/517925) on production, that is currently behind the `return_disabled_vulnerability_graphql_filters` feature flag. ## Owners - Most appropriate Slack channel to reach out to: `#g_srm_security_insights` - Best individual to reach out to: @wandering_person ## Expectations ### What are we expecting to happen? We should start seeing an array of disabled filters show up in vulnerabity-related graphql queries against groups that have more than 20,000 vulnerabilities. ### What can go wrong and how would we detect it? The main risks are that: - a bug could cause the graphql query to error (very low risk with this change) - The extra database query now happening in group vulnerability dashboard and security dashboard page could cause a performance regression or pages failing to load due to timeouts (low-medium risk) - team elastic dashboard: https://log.gprd.gitlab.net/app/dashboards#/view/a1f2bfe0-f15b-11ea-81e5-155ba78758d4?_g=h@3141a11 - sentry: https://new-sentry.gitlab.net/organizations/gitlab/discover/results/?field=title&field=event.type&field=project&field=user.display&field=timestamp&id=5&name=security+insights&project=3&project=18&query=&queryDataset=discover&sort=-timestamp&statsPeriod=7d&yAxis=count%28%29 ## Rollout Steps Note: Please make sure to run the chatops commands in the Slack channel that gets impacted by the command. ### Rollout on non-production environments - [x] Verify the MR with the feature flag is merged to `master` and has been deployed to non-production environments with ``` /chatops run auto_deploy status 00d8db977428ee819d2302cd39464ef88590cba8 ``` - [x] Enable the feature globally on non-production environments with ``` /chatops run feature set return_disabled_vulnerability_graphql_filters true --dev --pre --staging --staging-ref ``` - [x] Verify that the feature works as expected. - [x] If the feature flag causes end-to-end tests to fail, disable the feature flag on staging to avoid blocking [deployments](https://about.gitlab.com/handbook/engineering/deployments-and-releases/deployments/). - See [`#e2e-run-staging` Slack channel](https://gitlab.enterprise.slack.com/archives/CBS3YKMGD) and look for the following messages: - test kicked off: `Feature flag return_disabled_vulnerability_graphql_filters has been set to true on **gstg**` - test result: `This pipeline was triggered due to toggling of return_disabled_vulnerability_graphql_filters feature flag` For assistance with end-to-end test failures, please reach out via the [`#test-platform` Slack channel](https://gitlab.slack.com/archives/C3JJET4Q6). Note that end-to-end test failures on `staging-ref` [don't block deployments](https://about.gitlab.com/handbook/engineering/infrastructure/environments/staging-ref/#how-to-use-staging-ref). ### Specific rollout on production For visibility, all `/chatops` commands that target production must be executed in the [`#production` Slack channel](https://gitlab.slack.com/archives/C101F3796) and cross-posted (with the command results) to the responsible team's Slack channel. - [x] Ensure that the feature MRs have been deployed to both production and canary with ``` /chatops run auto_deploy status 00d8db977428ee819d2302cd39464ef88590cba8 ``` - [x] Depending on the [type of actor](https://docs.gitlab.com/ee/development/feature_flags/#feature-actors) you are using, pick one of these options: - For **project-actor**: `/chatops run feature set --project=gitlab-org/gitlab,gitlab-org/gitlab-foss,gitlab-com/www-gitlab-com return_disabled_vulnerability_graphql_filters true` - For **group-actor**: `/chatops run feature set --group=gitlab-org,gitlab-com return_disabled_vulnerability_graphql_filters true` - For **user-actor**: `/chatops run feature set --user=wandering_person return_disabled_vulnerability_graphql_filters true` - For **all internal users**: `/chatops run feature set --feature-group=gitlab_team_members return_disabled_vulnerability_graphql_filters true` - [x] Verify that the feature works for the specific actors. ### Preparation before global rollout - [x] Set a milestone to this rollout issue to signal for enabling and removing the feature flag when it is stable. - [x] Check if the feature flag change needs to be accompanied with a [change management issue](https://about.gitlab.com/handbook/engineering/infrastructure/change-management/#feature-flags-and-the-change-management-process). Cross link the issue here if it does. - no - [x] Ensure that you or a representative in development can be available for at least 2 hours after feature flag updates in production. If a different developer will be covering, or an exception is needed, please inform the oncall SRE by using the `@sre-oncall` Slack alias. - [x] Ensure that documentation exists for the feature, and the [version history text](https://docs.gitlab.com/development/documentation/feature_flags/#add-history-text) has been updated. - [x] Ensure that any breaking changes have been announced following the [release post process](https://about.gitlab.com/handbook/marketing/blog/release-posts/#deprecations-removals-and-breaking-changes) to ensure GitLab customers are aware. - [x] Notify the [`#support_gitlab-com` Slack channel](https://gitlab.slack.com/archives/C4XFU81LG) and your team channel ([more guidance when this is necessary in the dev docs](https://docs.gitlab.com/development/feature_flags/controls/#communicate-the-change)). ### Global rollout on production For visibility, all `/chatops` commands that target production must be executed in the [`#production` Slack channel](https://gitlab.slack.com/archives/C101F3796) and cross-posted (with the command results) to the responsible team's Slack channel. - [x] [Incrementally roll out](https://docs.gitlab.com/development/feature_flags/controls/#process) the feature on production. - Example: `/chatops run feature set return_disabled_vulnerability_graphql_filters <rollout-percentage> --actors`. - Between every step wait for at least 15 minutes and monitor the appropriate graphs on https://dashboards.gitlab.net. - [x] After the feature has been 100% enabled, wait for [at least one day before releasing the feature](#release-the-feature). ### Release the feature After the feature has been [deemed stable](https://about.gitlab.com/handbook/product-development-flow/feature-flag-lifecycle/#including-a-feature-behind-feature-flag-in-the-final-release), the [clean up](https://docs.gitlab.com/development/feature_flags/controls/#cleaning-up) should be done as soon as possible to permanently enable the feature and reduce complexity in the codebase. You can either [create a follow-up issue for Feature Flag Cleanup](https://gitlab.com/gitlab-org/gitlab/-/issues/new?issuable_template=Feature%20Flag%20Cleanup) or use the checklist below in this same issue. <!-- The checklist here is to help stakeholders keep track of the feature flag status --> - [x] Create a merge request to remove the `return_disabled_vulnerability_graphql_filters` feature flag. Ask for review/approval/merge as usual. The MR should include the following changes: - Remove all references to the feature flag from the codebase. - Remove the YAML definitions for the feature from the repository. - *MR:* https://gitlab.com/gitlab-org/gitlab/-/merge_requests/184617+s - [x] Ensure that the cleanup MR has been included in the release package. If the merge request was deployed before [the monthly release was tagged](https://about.gitlab.com/handbook/engineering/releases/#self-managed-releases-1), the feature can be officially announced in a release blog post: `/chatops run release check https://gitlab.com/gitlab-org/gitlab/-/merge_requests/180973 17.10` - [x] Close [the feature issue](https://gitlab.com/gitlab-org/gitlab/-/work_items/517925) to indicate the feature will be released in the current milestone. - [x] Once the cleanup MR has been deployed to production, clean up the feature flag from all environments by running these chatops command in `#production` channel: ``` /chatops run feature delete return_disabled_vulnerability_graphql_filters --dev --pre --staging --staging-ref --production ``` - [x] Close this rollout issue. ## ~Rollback Steps~ ~- [ ] This feature can be disabled on production by running the following Chatops command:~ ``` /chatops run feature set return_disabled_vulnerability_graphql_filters false ``` ~- [ ] Disable the feature flag on non-production environments:~ ``` /chatops run feature set return_disabled_vulnerability_graphql_filters false --dev --pre --staging --staging-ref ``` ~- [ ] Delete feature flag from all environments:~ ``` /chatops run feature delete return_disabled_vulnerability_graphql_filters --dev --pre --staging --staging-ref --production ```