[Feature flag] Enable geo_framework_verification

What

Remove the :geo_framework_verification feature flag.

By now, we should have validated and released :geo_package_file_verification. This issue releases verification for the rest of the Replicator classes.

Owners

  • Team: groupgeo
  • Most appropriate slack channel to reach out to: #g_geo
  • Best individual to reach out to: @mkozono

Expectations

What are we expecting to happen?

  • Geo primary begins backfilling checksums for (at the moment) Merge request diffs, Terraform states, Snippet repos
  • Geo secondaries begin verifying against those backfilled checksums
  • New replicables get checksummed and verified

What might happen if this goes wrong?

(GitLab.com production does not use Geo, but staging.gitlab.com does)

  • Nearly all verification logic is run in Sidekiq, so the most likely problem is probably exceptions in Sidekiq jobs
  • Related to that, perhaps some Sidekiq queue could overrun
  • If we really screwed up, then perhaps creation of new replicables could become blocked by exceptions

What can we monitor to detect problems with this?

Maybe https://dashboards.gitlab.net/d/sidekiq-main/sidekiq-overview?orgId=1

Beta groups/projects

N/A

Roll Out Steps

  • Find references to https://gitlab.com/gitlab-org/gitlab/-/issues/277400 in the codebase and make sure there are no TODOs like !45962 (diffs)
  • Enable on staging (/chatops run feature set geo_framework_verification true --staging)
  • Test on staging
  • Enable on GitLab Environment Toolkit 3k deployment with Geo
  • Test on 3k deployment
  • Ensure that documentation has been updated
  • [-] Enable on GitLab.com for individual groups/projects listed above and verify behaviour (/chatops run feature set --project=gitlab-org/gitlab feature_name true)
  • [-] Coordinate a time to enable the flag with the SRE oncall and release managers
    • In #production by pinging @sre-oncall
    • In #g_delivery by pinging @release-managers
  • [-] Announce on the issue an estimated time this will be enabled on GitLab.com
  • [-] Enable on GitLab.com by running chatops command in #production (/chatops run feature set feature_name true)
  • [-] Cross post chatops Slack command to #support_gitlab-com (more guidance when this is necessary in the dev docs) and in your team channel
  • [-] Announce on the issue that the flag has been enabled
  • Remove feature flag and add changelog entry
  • After the flag removal is deployed, clean up the feature flag by running chatops command in #production channel

Rollback Steps

  • This feature can be disabled on staging by running the following Chatops command:
/chatops run feature set geo_framework_verification false --staging
Edited by Michael Kozono