[Feature flag] Rollout of `geo_secondary_proxy` and `GEO_SECONDARY_PROXY`

What

Remove the :geo_secondary_proxy feature flag and the Workhorse environment variable GEO_SECONDARY_PROXY.

Description	Issue	MR(s)
Add endpoint /api/v4/geo/proxy	#324664 (closed)	!63343 (merged)
Determine if this is a Geo Proxy by calling /api/v4/geo/proxy	#329670 (closed)	!60769 (merged)
If this is a Geo Proxy, then use the Geo Proxy routing table	#329672 (closed)	!63321 (merged)
Cache the result of the request	#329671 (closed)	!66491 (merged)
[Feature flag] Rollout of geo_secondary_proxy and GEO_SECONDARY_PROXY	#325732 (closed) You are here

Owners

• Team: groupgeo
• Most appropriate slack channel to reach out to: #g_geo
• Best individual to reach out to: @mkozono

Expectations

What are we expecting to happen?

What might happen if this goes wrong?

What can we monitor to detect problems with this?

Beta groups/projects

If applicable, any groups/projects that are happy to have this feature turned on early. Some organizations may wish to test big changes they are interested in with a small subset of users ahead of time for example.

• gitlab-org/gitlab project
• gitlab-org/gitlab-com groups
• ...

Roll Out Steps

• Enable on staging (/chatops run feature set feature_name true --staging and set this environment variable GEO_SECONDARY_PROXY=1 on web nodes)
• Test on staging
• Get other teams to validate their features on https://geo.staging.gitlab.com => &6418 (closed)
• Ensure that documentation has been updated
• Enable on GitLab.com for individual groups/projects listed above and verify behaviour (/chatops run feature set --project=gitlab-org/gitlab feature_name true)
• If it is possible to perform an incremental rollout, this should be preferred. Proposed increments are: 10%, 50%, 100%. Proposed minimum time between increments is 15 minutes.
• Coordinate a time to enable the flag with the SRE oncall and release managers
  • In #production mention @sre-oncall and @release-managers. Once an SRE on call and Release Manager on call confirm, you can proceed with the rollout
• Announce on the issue an estimated time this will be enabled on GitLab.com. Note: Once a feature rollout has started, it is not necessary to inform @sre-oncall/@release-managers at each stage of the gradual rollout.
• Enable on GitLab.com by running chatops command in #production (/chatops run feature set feature_name true)
• Cross post chatops Slack command to #support_gitlab-com (more guidance when this is necessary in the dev docs) and in your team channel
• Announce on the issue that the flag has been enabled
• Remove feature flag and add changelog entry. Ensure that the feature flag definition YAML file has been removed in the same MR that is removing the feature flag from the code
• Remove the Workhorse environment variable check for GEO_SECONDARY_PROXY=1 (Various unrelated Workhorse tests will need to be fixed)
• Revert https://gitlab.com/gitlab-com/gl-infra/chef-repo/-/merge_requests/834 if GEO_SECONDARY_PROXY=1 is no longer needed due to inverting the default
• After the flag removal is deployed, clean up the feature flag by running chatops command in #production channel

Rollback Steps

• This feature can be disabled by running the following Chatops command:

/chatops run feature set --project=gitlab-org/gitlab feature_name false