<!-- Title suggestion: [Feature flag] Enable description of feature --> ## Summary This issue is to rollout [the async chain of custody report](https://gitlab.com/gitlab-org/gitlab/-/issues/342594) on production, that is currently behind the `async_chain_of_custody_report` feature flag. <!-- Short description of what the feature is about and link to relevant other issues. --> The existing behavior when requesting a chain of custody csv is that the report is generated in the controller and a CSV is immediately downloaded in the user's browser. The async report will be generated in a background job, which fires of an email with the CSV attached to the requesting user. ## Owners - Team: @gitlab-org/manage/compliance - Most appropriate slack channel to reach out to: `#g_manage_compliance` - Best individual to reach out to: Michael Becker (@wandering_person) - PM: @stkerr ## Expectations ### What are we expecting to happen? we enable to feature on a group-by-group basis, we see the same report being obtained via email rather than direct download ### What might happen if this goes wrong? We would get customer contacts that they cannot obtain their report. ### What can we monitor to detect problems with this? <!-- Which dashboards from https://dashboards.gitlab.net are most relevant? --> _Consider mentioning checks for 5xx errors or other anomalies like an increase in redirects (302 HTTP response status)_ ### What can we check for monitoring production after rollouts? _Consider adding links to check for Sentry errors, Production logs for 5xx, 302s, etc._ ## Rollout Steps ### Rollout on non-production environments - Ensure that the feature MRs have been deployed to non-production environments. - [x] `/chatops run auto_deploy status <merge-commit-of-your-feature>` - [x] Enable the feature globally on non-production environments. - [x] `/chatops run feature set <feature-flag-name> true --dev --staging --staging-ref` - [x] Verify that the feature works as expected. Posting the QA result in this issue is preferable. The best environment to validate the feature in is [staging-canary](https://about.gitlab.com/handbook/engineering/infrastructure/environments/#staging-canary) as this is the first environment deployed to. Note you will need to make sure you are configured to use canary as outlined [here](https://about.gitlab.com/handbook/engineering/infrastructure/environments/canary-stage/) when accessing the staging environment in order to make sure you are testing appropriately. ### Specific rollout on production - Ensure that the feature MRs have been deployed to both production and canary. - [x] `/chatops run auto_deploy status <merge-commit-of-your-feature>` - If you're using [project-actor](https://docs.gitlab.com/ee/development/feature_flags/#feature-actors), you must enable the feature on these entries: - [x] `/chatops run feature set --project=gitlab-org/gitlab,gitlab-org/gitlab-foss,gitlab-com/www-gitlab-com <feature-flag-name> true` - If you're using [group-actor](https://docs.gitlab.com/ee/development/feature_flags/#feature-actors), you must enable the feature on these entries: - [x] `/chatops run feature set --group=gitlab-org,gitlab-com <feature-flag-name> true` - If you're using [user-actor](https://docs.gitlab.com/ee/development/feature_flags/#feature-actors), you must enable the feature on these entries: - [x] `/chatops run feature set --user=<your-username> <feature-flag-name> true` - [x] Verify that the feature works on the specific entries. Posting the QA result in this issue is preferable. ### Preparation before global rollout - [x] Check if the feature flag change needs to be accompanied with a [change management issue](https://about.gitlab.com/handbook/engineering/infrastructure/change-management/#feature-flags-and-the-change-management-process). Cross link the issue here if it does. - [x] Ensure that you or a representative in development can be available for at least 2 hours after feature flag updates in production. If a different developer will be covering, or an exception is needed, please inform the oncall SRE by using the `@sre-oncall` Slack alias. - [ ] Ensure that documentation has been updated ([More info](https://docs.gitlab.com/ee/development/documentation/feature_flags.html#features-that-became-enabled-by-default)). - [x] Announce on [the feature issue](ISSUE LINK) an estimated time this will be enabled on GitLab.com. - [x] Ensure that any breaking changes have been announced following the [release post process](https://about.gitlab.com/handbook/marketing/blog/release-posts/#deprecations-removals-and-breaking-changes) to ensure GitLab customers are aware. - [x] Notify `#support_gitlab-com` and your team channel ([more guidance when this is necessary in the dev docs](https://docs.gitlab.com/ee/development/feature_flags/controls.html#communicate-the-change)). ### Global rollout on production For visibility, all `/chatops` commands that target production should be executed in the `#production` slack channel and cross-posted (with the command results) to the responsible team's slack channel (`#g_TEAM_NAME`). - [x] [Incrementally roll out](https://docs.gitlab.com/ee/development/feature_flags/controls.html#process) the feature. - If the feature flag in code has [an actor](https://docs.gitlab.com/ee/development/feature_flags/#feature-actors), perform **actor-based** rollout. - [x] `/chatops run feature set <feature-flag-name> <rollout-percentage> --actors` - If the feature flag in code does **NOT** have [an actor](https://docs.gitlab.com/ee/development/feature_flags/#feature-actors), perform time-based rollout (**random** rollout). - [x] `/chatops run feature set <feature-flag-name> <rollout-percentage> --random` - Enable the feature globally on production environment. - [x] `/chatops run feature set <feature-flag-name> true` - [x] Announce on [the feature issue](ISSUE LINK) that the feature has been globally enabled. - [ ] Wait for [at least one day for the verification term](https://about.gitlab.com/handbook/product-development-flow/feature-flag-lifecycle/#including-a-feature-behind-feature-flag-in-the-final-release). ### Release the feature After the feature has been [deemed stable](https://about.gitlab.com/handbook/product-development-flow/feature-flag-lifecycle/#including-a-feature-behind-feature-flag-in-the-final-release), the [clean up](https://docs.gitlab.com/ee/development/feature_flags/controls.html#cleaning-up) should be done as soon as possible to permanently enable the feature and reduce complexity in the codebase. You can either [create a follow-up issue for Feature Flag Cleanup](https://gitlab.com/gitlab-org/gitlab/-/issues/new?issuable_template=Feature%20Flag%20Cleanup) or use the checklist below in this same issue. <!-- The checklist here is to help stakeholders keep track of the feature flag status --> - [x] Create a merge request to remove `<feature-flag-name>` feature flag. Ask for review and merge it. - [x] Remove all references to the feature flag from the codebase. - [x] Remove the YAML definitions for the feature from the repository. - [x] Create [a changelog entry](https://docs.gitlab.com/ee/development/feature_flags/#changelog). - [ ] Ensure that the cleanup MR has been included in the release package. If the merge request was deployed before [the monthly release was tagged](https://about.gitlab.com/handbook/engineering/releases/#self-managed-releases-1), the feature can be officially announced in a release blog post. - [ ] `/chatops run release check <merge-request-url> <milestone>` - [ ] Close [the feature issue](ISSUE LINK) to indicate the feature will be released in the current milestone. - [ ] If not already done, clean up the feature flag from all environments by running these chatops command in `#production` channel: - [ ] `/chatops run feature delete <feature-flag-name> --dev --staging --staging-ref --production` - [ ] Close this rollout issue. ## Rollback Steps - [ ] This feature can be disabled by running the following Chatops command: ``` /chatops run feature set <feature-flag-name> false ```