Enable automatic optimizing of batch sizes and other parameters of batched migrations
Feature
This feature uses the :optimize_batched_migrations
feature flag!
Once enabled, this feature automatically adjusts batch sizes of batched background migrations. This is based on the history of jobs and used to adapt to workload automatically. The goal is to increase throughput of batched background migrations.
Owners
- Team: groupdatabase
- Most appropriate slack channel to reach out to:
#g_database
- Best individual to reach out to: @abrandl @iroussos
- PM: @fabian
The Rollout Plan
This is primarily being used for primary key migrations on .com. groupdatabase to enable/disable and tune as needed.
Expectations
What are we expecting to happen?
Increase throughput of batched background migrations
What might happen if this goes wrong?
We only risk decreasing throughput, we don't overload the system
What can we monitor to detect problems with this?
tbd
Rollout Timeline
tbd
Initial Rollout
Preparation Phase
-
Enable on staging ( /chatops run feature set feature_name true --staging
) -
Test on staging -
Ensure that documentation has been updated (More info) -
Announce on the issue an estimated time this will be enabled on GitLab.com
Partial Rollout Phase
-
Enable on GitLab.com for individual groups/projects listed above and verify behaviour ( /chatops run feature set --project=gitlab-org/gitlab feature_name true
) -
Verify behaviour (See Beta Groups) and add details with screenshots as a comment on this issue -
If it is possible to perform an incremental rollout, this should be preferred. Proposed increments are: 10%
,50%
,100%
. Proposed minimum time between increments is 15 minutes.- When setting percentages, make sure that the feature works correctly between feature checks. See #327117 (closed) for more information
- For actor-based rollout:
/chatops run feature set feature_name 10 --actors
- For time-based rollout:
/chatops run feature set feature_name 10
-
Make the feature flag enabled by default i.e. Change default_enabled
totrue
-
Cross post chatops slack command to #support_gitlab-com
(more guidance when this is necessary in the dev docs) and in your team channel
Cleanup
This is an important phase, that should be either done in the next Milestone or as soon as possible. For the cleanup phase, please follow our documentation on how to clean up the feature flag.
-
Announce on the issue that the flag has been enabled -
Remove :feature_name
feature flag-
Remove all references to the feature flag from the codebase -
Remove the YAML definitions for the feature from the repository -
Create a Changelog Entry
-
-
Clean up the feature flag from all environments by running this chatops command in #production
channel/chatops run feature delete some_feature
.
Final Step
-
Close this rollout issue for the feature flag after the feature flag is removed from the codebase.
Rollback Steps
-
This feature can be disabled by running the following Chatops command:
/chatops run feature set --project=gitlab-org/gitlab feature_name false