<!-- Title suggestion: [Feature flag] Enable description of feature --> # Summary <!-- Short description of what the feature is about and link to relevant other issues. --> This issue is to rollout [Improve performance of SubscriptionSynchronizer to avoid timeouts](https://gitlab.com/gitlab-org/customers-gitlab-com/-/work_items/13202) on CustomersDot production, that is currently behind the [`zuora_fetch_outside_transaction`](https://gitlab.com/gitlab-org/customers-gitlab-com/-/feature_flags/405/edit) feature flag. <!-- Note: Feature flags should be implemented using Unleash (e.g. `Unleash.enabled?(:flag_name)`), not Flipper (e.g. `Feature.enabled?(:flag_name)`). --> ## Owners - Team: Fulfillment Platform - Most appropriate Slack channel to reach out to: `#g_fulfillment_platform` - Best individual to reach out to: `@vshumilo` - PM: `TBD` ## Stakeholders - `#g_fulfillment_platform` team ## The Rollout Plan - Rollout on CustomersDot staging for validation - Rollout Feature for everyone as soon as it's ready <!-- Which dashboards from https://dashboards.gitlab.net are most relevant? Sentry errors reports can also be useful to review --> ## Expectations ### What are we expecting to happen? Zuora Subscription and Subscription related records (Rate Plans, Rate Plan Charges and Rate Plan Charge Tiers) Sync outside DB transactions to avoid timeouts. ### What might happen if this goes wrong? Subscriptions will fail to sync this might block actions on the subscriptions and provisioning. ### What can we monitor to detect problems with this? - GCP - Provision monitoring - We also should complete validation for subscription sync in staging ## Rollout Steps ### Rollout on non-production environments - [x] Ensure that the feature MRs have been deployed to non-production environments, e.g. `customers-staging` and `customers-staging-ref`. - [x] Consider announcing that the flag will be enabled to potentially impacted groups, e.g. `#s_fulfillment_engineering` - [x] Enable the feature globally on non-production environments by applying the appropriate environment, e.g. `customers-development`, `customers-staging` and `customers-staging-ref`. - When enabling the feature flag for `customers-staging`, it should also be enabled on `customers-staging-ref` at the same time. - [x] After enabling the feature flag for `customers-staging`, monitor the E2E test results in `#s_fulfillment_status` or `#e2e-run-staging` for any related failures. - [x] Verify that the feature works as expected. Posting the QA result in this issue is preferable. ### Preparation before production rollout - [ ] ~Consider creating an issue to align with stakeholders and cross-functional groups on a production rollout date ([example](https://gitlab.com/gitlab-org/customers-gitlab-com/-/issues/6764)).~ - [ ] Stub the feature flag as enabled by default [in RSpec](https://gitlab.com/gitlab-org/customers-gitlab-com/-/blob/a5cc41ad82db8ba80f44ee6cf21d870451a88882/spec/rails_helper.rb#L83) to avoid unexpected side-effects and ensure better coverage. - [ ] Ensure that the feature MRs have been deployed to production - [ ] ~Check if the feature flag change needs to be accompanied with a [change management issue](https://about.gitlab.com/handbook/engineering/infrastructure/change-management/#feature-flags-and-the-change-management-process).~ Cross link the issue here if it does. - [ ] Ensure that you or a representative in Fulfillment can be available for at least 2 hours after feature flag updates in production. If a different developer will be covering, or an exception is needed, please inform the Fulfillment team in `#s_fulfillment_engineering` - [ ] Ensure any documentation has been updated - [ ] Announce on [the feature issue](https://gitlab.com/gitlab-org/customers-gitlab-com/-/work_items/13202) an estimated time this will be enabled on production if applicable - [ ] If the feature might impact the user experience, notify `#s_fulfillment`, `#support_licensing-subscription` and potentially `#support_gitlab-com`, as well as your team channel ([more guidance when this is necessary in the dev docs](https://docs.gitlab.com/development/feature_flags/controls/#communicate-the-change)). ### Global rollout on CustomersDot production - [ ] Confirm the feature flag is enabled on `staging` without incident - [ ] Validate the change is working as expected in `staging`. - [ ] Enable the feature globally on production environment - [ ] Announce on [the feature issue](https://gitlab.com/gitlab-org/customers-gitlab-com/-/work_items/13202) that the feature has been globally enabled. - [ ] Wait for [at least one day for the verification term](https://about.gitlab.com/handbook/product-development-flow/feature-flag-lifecycle/#including-a-feature-behind-feature-flag-in-the-final-release). ### Release the feature After the feature has been [deemed stable](https://about.gitlab.com/handbook/product-development-flow/feature-flag-lifecycle/#including-a-feature-behind-feature-flag-in-the-final-release), the clean up should be done as soon as possible to permanently enable the feature and reduce complexity in the codebase. <!-- The checklist here is to help stakeholders keep track of the feature flag status --> - [ ] Create a merge request to remove `zuora_fetch_outside_transaction` feature flag - [ ] Remove all references to the feature flag from the codebase - [ ] Remove references under `/qa` directory if applicable - [ ] Ensure that the cleanup MR has been deployed to staging, staging-ref and production - [ ] Disable the feature flag via the [CustomersDot dashboard](https://gitlab.com/gitlab-org/customers-gitlab-com/-/feature_flags) - [ ] Ensure production environment is still working as expected - [ ] Remove the Unleash feature flag from the [CustomersDot dashboard](https://gitlab.com/gitlab-org/customers-gitlab-com/-/feature_flags) - [ ] Close this rollout issue. ## Rollback Steps - [ ] This feature can be disabled by performing the following actions: - [ ] Disable the Unleash flag on the [feature flag page](https://gitlab.com/gitlab-org/customers-gitlab-com/-/feature_flags/405/edit) /label ~"devops::fulfillment" ~"section::fulfillment" ~"feature flag" <!-- markdownlint-disable MD044 --> /assign `DRI`