## Summary This issue is to roll out the feature on production, which is currently behind the `auto_enable_restricted_access_on_self_managed` feature flag. This feature flag controls the automatic force-enabling of restricted access on Self-Managed instances when updating the `Contract overages allowed` custom attribute in Zuora for a license. It is used in `GitlabSubscriptions::UpdateLicenseDependenciesService`. ## Owners * Most appropriate Slack channel to reach out to: `#g_seat_management` * Best individual to reach out to: @lwanko ## Expectations ### What are we expecting to happen? When the `Contract overages allowed` custom attribute is updated in Zuora for a Self-Managed license, restricted access (seat control block overages) is automatically force-enabled on the corresponding Self-Managed instance without requiring manual intervention. ### What can go wrong and how would we detect it? * Instances where overages should still be allowed could be incorrectly blocked. * Monitor error rates in `GitlabSubscriptions::UpdateLicenseDependenciesService` and watch for support tickets from Self-Managed customers reporting unexpected seat blocking behavior. ## Rollout Steps ### Rollout on non-production environments * Verify the MR with the feature flag is merged to `master` and has been deployed to non-production environments with `/chatops gitlab run auto_deploy status <merge-commit-of-your-feature>` * Deploy the feature flag at a percentage (recommended percentage: 50%) with `/chatops gitlab run feature set auto_enable_restricted_access_on_self_managed 50 --actors --dev --pre --staging --staging-ref` * Monitor that the error rates did not increase (repeat with a different percentage as necessary). * Enable the feature globally on non-production environments with `/chatops gitlab run feature set auto_enable_restricted_access_on_self_managed true --dev --pre --staging --staging-ref` * Verify that the feature works as expected. The best environment to validate the feature in is `staging-canary` as this is the first environment deployed to. Make sure you are `configured to use canary`. * If the feature flag causes end-to-end tests to fail, disable the feature flag on staging to avoid blocking `deployments`. * See `#e2e-run-staging Slack channel` and look for the following messages: * test kicked off: `Feature flag auto_enable_restricted_access_on_self_managed has been set to true on **gstg**` * test result: `This pipeline was triggered due to toggling of auto_enable_restricted_access_on_self_managed feature flag` If you encounter end-to-end test failures and are unable to diagnose them, you may reach out to the `#s_developer_experience Slack channel` for assistance. ### Before production rollout * If the change is significant and you wanted to announce in `#whats-happening-at-gitlab`, it best to do it before rollout to `gitlab-org/gitlab-com`. ### Specific rollout on production For visibility, all `/chatops` commands that target production must be executed in the `#production Slack channel` and cross-posted (with the command results) to the responsible team's Slack channel. * Ensure that the feature MRs have been deployed to both production and canary with `/chatops gitlab run auto_deploy status <merge-commit-of-your-feature>` * For **all internal users**: `/chatops gitlab run feature set --feature-group=gitlab_team_members auto_enable_restricted_access_on_self_managed true` * Verify that the feature works for the specific actors. ### Preparation before global rollout * Set a milestone to this rollout issue to signal for enabling and removing the feature flag when it is stable. * Check if the feature flag change needs to be accompanied with a `change management issue`. Cross link the issue here if it does. * Ensure that you or a representative in development can be available for at least 2 hours after feature flag updates in production. * Ensure that documentation exists for the feature, and the [version history text](https://docs.gitlab.com/development/documentation/feature_flags/#add-history-text) has been updated. * Notify the `#support_gitlab-com Slack channel` and your team channel. ### Global rollout on production For visibility, all `/chatops` commands that target production must be executed in the `#production Slack channel` and cross-posted (with the command results) to the responsible team's Slack channel. * [ ] [Incrementally roll out](https://docs.gitlab.com/development/feature_flags/controls/#process) the feature on production. * [ ] Example: `/chatops gitlab run feature set auto_enable_restricted_access_on_self_managed <rollout-percentage> --actors`. * Between every step wait for at least 15 minutes and monitor the appropriate graphs on `https://dashboards.gitlab.net`. * [ ] After the feature has been 100% enabled, wait for at least one day before releasing the feature. ### Release the feature * [ ] Create a merge request to remove the `auto_enable_restricted_access_on_self_managed` feature flag. The MR should include: * [ ] Remove all references to the feature flag from the codebase. * Remove the YAML definitions for the feature from the repository. * [ ] Ensure that the cleanup MR has been included in the release package. * [ ] Close the feature issue to indicate the feature will be released in the current milestone. * [ ] Once the cleanup MR has been deployed to production, clean up the feature flag from all environments: `/chatops gitlab run feature delete auto_enable_restricted_access_on_self_managed --dev --pre --staging --staging-ref --production` * [ ] Close this rollout issue. ## Rollback Steps * Disable on production: Insert at cursor `/chatops gitlab run feature set auto_enable_restricted_access_on_self_managed false` * Disable on non-production environments: Insert at cursor `/chatops gitlab run feature set auto_enable_restricted_access_on_self_managed false --dev --pre --staging --staging-ref` * Delete feature flag from all environments: Insert at cursor `/chatops gitlab run feature delete auto_enable_restricted_access_on_self_managed --dev --pre --staging --staging-ref --production`