[Feature flag] Enable Geo replication of LFS objects by the new framework
This feature uses the
:geo_lfs_object_replication feature flag!
- Most appropriate slack channel to reach out to:
- Best individual to reach out to: @ibaum
- PM: @fzimmer
The Rollout Plan
- There is no rollout on production gitlab.com since Geo is not enabled there
What are we expecting to happen?
When an LFS object is created, there should be less delay before Geo secondary sites make a request to download it. Otherwise, no change.
What might happen if this goes wrong?
If something goes wrong, then:
- Contact #g_geo in Slack
- Turn off the feature flag
Some things that could go wrong:
- LFS objects are not replicated to Geo secondary sites
- Progress bar for LFS object replication on /admin/geo/nodes shows less than near 100% synced
- Sentry errors on primary or secondary sites
What can we monitor to detect problems with this?
There should be no impact on production gitlab.com.
On staging, check:
Enable on staging (
/chatops run feature set geo_lfs_object_replication true --staging)
Test on staging
Test on GCP, e.g. using GET 3k + 3k Geo deployments
Ensure that documentation has been updated (More info)
[-] Announce on the issue an estimated time this will be enabled on GitLab.com
[-] Check if the feature flag change needs to be accompagnied with a change management issue. Cross link the issue here if it does.
Partial Rollout Phase
[-] Enable on GitLab.com for individual groups/projects listed above and verify behaviour (
/chatops run feature set --project=gitlab-org/gitlab feature_name true)
[-] Verify behaviour (See Beta Groups) and add details with screenshots as a comment on this issue
[-] If it is possible to perform an incremental rollout, this should be preferred. Proposed increments are:
100%. Proposed minimum time between increments is 15 minutes.
- When setting percentages, make sure that the feature works correctly between feature checks. See #327117 (closed) for more information
- For actor-based rollout:
/chatops run feature set feature_name 10 --actors
- For time-based rollout:
/chatops run feature set feature_name 10
Make the feature flag enabled by default i.e. Change
Cross post chatops slack command to
#support_gitlab-com(more guidance when this is necessary in the dev docs) and in your team channel
This is an important phase, that should be either done in the next Milestone or as soon as possible. For the cleanup phase, please follow our documentation on how to clean up the feature flag.
Announce on the issue that the flag has been enabled
Remove all references to the feature flag from the codebase
Remove the YAML definitions for the feature from the repository
Create a Changelog Entry
Clean up the feature flag from all environments by running this chatops command in
/chatops run feature delete geo_lfs_object_replication.
Close this rollout issue for the feature flag after the feature flag is removed from the codebase.
This feature can be disabled by running the following Chatops command:
/chatops run feature set geo_lfs_object_replication false