## Summary Roll out [the feature](https://gitlab.com/gitlab-org/gitlab/-/issues/296977) currently behind the `updated_gpg_key_email` feature flag. - DRI: to be named by the group. I wrote the change as a community contributor, so I cannot run the chatops commands below or add the `feature flag` label. - Team Slack channel: `#g_create_repository-services` > [!note] > Process and guidance live in the docs. This issue is just the commands and a place to track the rollout. > "Rolling out" means incrementally enabling the flag on GitLab.com to validate stability. It is not the same as releasing the feature, which happens when the flag is removed. > [Feature flag controls](https://docs.gitlab.com/development/feature_flags/controls/), [Feature flag lifecycle](https://handbook.gitlab.com/handbook/product-development/how-we-work/product-development-flow/feature-flag-lifecycle/#feature-flag-lifecycle) ## What could go wrong? The flag only decides whether `Repositories::GpgKeys::UpdateService` enqueues the new `updated_gpg_key_email` email, which tells the owner that an update added a subkey or a user ID to their GPG key. The update of the key is not behind the flag. If the email fails, the owner of an updated key gets no email, and nothing else changes. The flag exists because `doc/development/emails.md` asks for a new mailer method to be behind one, so that it is not enqueued before every Sidekiq node has it. ## Events - [Exceptions with updated_gpg_key_email:1](https://log.gprd.gitlab.net/app/discover#/?_g=(time:(from:now-1d,to:now))&_a=(index:'7092c4e2-4eb5-46f2-8305-a7da2edad090',query:(language:kuery,query:'json.exception.feature_flag_states.keyword:%22updated_gpg_key_email:1%22'),columns:!(json.exception.class,json.exception.message,json.extra.workflow_id,json.meta.caller_id),sort:!(!(json.time,desc)))) - [Events with updated_gpg_key_email:1](https://log.gprd.gitlab.net/app/discover#/?_g=(time:(from:now-1d,to:now))&_a=(index:'7092c4e2-4eb5-46f2-8305-a7da2edad090',query:(language:kuery,query:'json.feature_flag_states:%22updated_gpg_key_email:1%22'),columns:!(json.meta.caller_id,json.feature_flag_states),sort:!(!(json.time,desc)))) - [Error rate and other graphs by modifying the examples in the Visualization Library](https://log.gprd.gitlab.net/app/visualize#/?_g=h@358d019&s=FF%20Observability) Feature Flag events are only logged by default for feature flags marked for the current or future milestones. To enable while the feature flag is active, see https://docs.gitlab.com/development/feature_flags/#logging ## Rollout Run all production `/chatops` in [`#production`](https://gitlab.slack.com/archives/C101F3796) and cross-post the results to `#g_create_repository-services`. Background: [incremental rollout process](https://docs.gitlab.com/development/feature_flags/controls/#process), [feature actors](https://docs.gitlab.com/development/feature_flags/#feature-actors). **Non-production** ``` /chatops gitlab run feature set updated_gpg_key_email 50 --actors --dev --pre --staging --staging-ref /chatops gitlab run feature set updated_gpg_key_email true --dev --pre --staging --staging-ref ``` **Production**: percentage rollout (wait at least 15 minutes between steps, watch dashboards): ``` /chatops gitlab run feature set updated_gpg_key_email <percentage> --actors ``` Or target specific actors instead: ``` /chatops gitlab run feature set --project=gitlab-org/gitlab,gitlab-org/gitlab-foss updated_gpg_key_email true /chatops gitlab run feature set --group=gitlab-org,gitlab-com updated_gpg_key_email true /chatops gitlab run feature set --user=<gitlab-username-of-dri> updated_gpg_key_email true ``` ## Before global rollout Confirm the relevant gotchas before going to 100%. See [enabling a feature for GitLab.com](https://docs.gitlab.com/development/feature_flags/controls/#enabling-a-feature-for-gitlabcom): - [Docs + version history](https://docs.gitlab.com/development/documentation/feature_flags/) updated - [Breaking changes](https://docs.gitlab.com/development/documentation/release_notes/#deprecations-removals-and-breaking-changes) announced, if any - [Change management issue](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/#feature-flags-and-the-change-management-process) opened, if required - [External API consumers](https://docs.gitlab.com/development/feature_flags/#do-not-use-feature-flags-in-external-api-consumers) handled with a fail-open mechanism, if applicable ## Cleanup Remove the flag once [deemed stable](https://handbook.gitlab.com/handbook/product-development/how-we-work/product-development-flow/feature-flag-lifecycle/#feature-flag-lifecycle). See [cleaning up](https://docs.gitlab.com/development/feature_flags/controls/#cleaning-up). Track it here, or open a follow-up [Feature Flag Cleanup issue](https://gitlab.com/gitlab-org/gitlab/-/work_items/new?description_template=Feature%20Flag%20Cleanup). Remove the flag and its YAML definition from the codebase, then: ``` /chatops gitlab run release check <merge-request-url> <milestone> /chatops gitlab run feature delete updated_gpg_key_email --dev --pre --staging --staging-ref --production ``` ## Rollback ``` /chatops gitlab run feature set updated_gpg_key_email false # production /chatops gitlab run feature set updated_gpg_key_email false --dev --pre --staging --staging-ref # non-production /chatops gitlab run feature delete updated_gpg_key_email --dev --pre --staging --staging-ref --production # remove entirely ```