[Feature flag] Enable geo_skip_download_if_exists
Summary
This issue is to roll out the feature on production,
that is currently behind the geo_skip_download_if_exists
feature flag.
Owners
- Most appropriate Slack channel to reach out to:
#g_geo
- Best individual to reach out to: @mkozono
Expectations
What are we expecting to happen?
On GitLab.com, staging.gitlab.com, nothing.
On most GitLab deployments with Geo enabled (e.g. staging-ref.gitlab.com), nothing.
This code is not triggered during normal operation.
It can be triggered in scenarios like:
- you did a planned failover and attached the old primary as a secondary without rebuilding it.
- you did a failover test on a secondary, rewound the PG DB, and reattached it without rebuilding it.
- you restored a backup and attached the site as a secondary
- you manually copied a replicable to a secondary site to workaround a sync problem
- you reset the Geo tracking database to workaround a problem
If a blob/file already exists on a secondary site when the secondary site attempts to replicate it for the first time (according to the Geo tracking database), then the site will skip downloading the file. Instead, it will be marked as already synced. Soon after that, a background job will verify that the file is identical to the file that the primary has.
What can go wrong and how would we detect it?
- If the conditional logic is written incorrectly, then worst case, blobs become marked synced without being synced. This would then manifest as an increase in verification and sync failures for blobs.
- If the conditional logic raises an error, then there would be a persistent decrease in the percentage of successfully synced blobs.
We can check Admin UI > Geo > Sites.
Rollout Steps
Note: Please make sure to run the chatops commands in the Slack channel that gets impacted by the command.
Rollout on staging-ref
- Verify the MR with the feature flag is merged to
master
and have been deployed to non-production environments with/chatops run auto_deploy status <merge-commit-of-your-feature>
-
Enable the feature on staging-ref /chatops run feature set geo_skip_download_if_exists true --staging-ref
-
Verify that the feature works as expected. -
Ensure that documentation exists for the feature, and the version history text has been updated. => !140276 (merged) -
Leave a comment on the feature issue announcing that the feature has been enabled. -
Wait for at least one day for the verification term.
Release the feature
After the feature has been deemed stable, the clean up should be done as soon as possible to permanently enable the feature and reduce complexity in the codebase.
You can either create a follow-up issue for Feature Flag Cleanup or use the checklist below in this same issue.
-
Create a merge request to remove the geo_skip_download_if_exists
feature flag. Ask for review/approval/merge as usual. The MR should include the following changes:- Remove all references to the feature flag from the codebase.
- Remove the YAML definitions for the feature from the repository.
- Create a changelog entry.
- Here it is: !140276 (merged)
-
Ensure that the cleanup MR has been included in the release package. If the merge request was deployed before the monthly release was tagged, the feature can be officially announced in a release blog post: /chatops run release check <merge-request-url> <milestone>
-
Close the feature issue to indicate the feature will be released in the current milestone. -
Clean up the feature flag from staging-ref by running these chatops command in #production
channel:/chatops run feature delete geo_skip_download_if_exists --staging-ref
-
Close this rollout issue.
Rollback Steps
-
This feature can be disabled by running the following Chatops command:
/chatops run feature set geo_skip_download_if_exists false