[Feature flag] Rollout of `geo_secondary_proxy` and `GEO_SECONDARY_PROXY`
<!-- Title suggestion: [Feature flag] Enable description of feature -->
## What
Remove the `:geo_secondary_proxy` feature flag and the Workhorse environment variable `GEO_SECONDARY_PROXY`.
| Description | Issue | MR(s) |
| --- | --- | --- |
| Add endpoint `/api/v4/geo/proxy` | https://gitlab.com/gitlab-org/gitlab/-/issues/324664 | https://gitlab.com/gitlab-org/gitlab/-/merge_requests/63343 |
| Determine if this is a Geo Proxy by calling `/api/v4/geo/proxy` | https://gitlab.com/gitlab-org/gitlab/-/issues/329670 | https://gitlab.com/gitlab-org/gitlab/-/merge_requests/60769 |
| If this is a Geo Proxy, then use the Geo Proxy routing table | https://gitlab.com/gitlab-org/gitlab/-/issues/329672 | https://gitlab.com/gitlab-org/gitlab/-/merge_requests/63321 |
| Cache the result of the request | https://gitlab.com/gitlab-org/gitlab/-/issues/329671 | https://gitlab.com/gitlab-org/gitlab/-/merge_requests/66491 |
| [Feature flag] Rollout of `geo_secondary_proxy` and `GEO_SECONDARY_PROXY` | https://gitlab.com/gitlab-org/gitlab/-/issues/325732 **You are here** | |
## Owners
- Team: ~"group::geo"
- Most appropriate slack channel to reach out to: `#g_geo`
- Best individual to reach out to: @mkozono
## Expectations
### What are we expecting to happen?
### What might happen if this goes wrong?
### What can we monitor to detect problems with this?
<!-- Which dashboards from https://dashboards.gitlab.net are most relevant? Sentry errors reports can also be useful to review -->
## Beta groups/projects
If applicable, any groups/projects that are happy to have this feature turned on early. Some organizations may wish to test big changes they are interested in with a small subset of users ahead of time for example.
- `gitlab-org/gitlab` project
- `gitlab-org`/`gitlab-com` groups
- ...
## Roll Out Steps
- [x] Enable on staging (`/chatops run feature set feature_name true --staging` and set this environment variable `GEO_SECONDARY_PROXY=1` on web nodes)
- [x] Test on staging
- [ ] Get other teams to validate their features on https://geo.staging.gitlab.com => https://gitlab.com/groups/gitlab-org/-/epics/6418
- [x] Ensure that documentation has been updated
- [ ] ~~Enable on GitLab.com for individual groups/projects listed above and verify behaviour (`/chatops run feature set --project=gitlab-org/gitlab feature_name true`)~~
- [ ] ~~If it is possible to perform an incremental rollout, this should be preferred. Proposed increments are: `10%`, `50%`, `100%`. Proposed minimum time between increments is 15 minutes.~~
- [ ] ~~Coordinate a time to enable the flag with the SRE oncall and release managers~~
- ~~In `#production` mention `@sre-oncall` and `@release-managers`. Once an SRE on call and Release Manager on call confirm, you can proceed with the rollout~~
- [ ] ~~Announce on the issue an estimated time this will be enabled on GitLab.com. **Note**: Once a feature rollout has started, it is not necessary to inform `@sre-oncall`/`@release-managers` at each stage of the gradual rollout.~~
- [ ] ~~Enable on GitLab.com by running chatops command in `#production` (`/chatops run feature set feature_name true`)~~
- [ ] ~~Cross post chatops Slack command to `#support_gitlab-com` ([more guidance when this is necessary in the dev docs](https://docs.gitlab.com/ee/development/feature_flags/controls.html#where-to-run-commands)) and in your team channel~~
- [ ] ~~Announce on the issue that the flag has been enabled~~
- [x] Remove feature flag and add changelog entry. Ensure that the feature flag definition YAML file has been removed in the **same MR** that is removing the feature flag from the code
- [x] Remove the Workhorse environment variable check for `GEO_SECONDARY_PROXY=1` (Various unrelated Workhorse tests will need to be fixed)
- [ ] Revert https://gitlab.com/gitlab-com/gl-infra/chef-repo/-/merge_requests/834 if `GEO_SECONDARY_PROXY=1` is no longer needed due to inverting the default
- [x] After the flag removal is deployed, [clean up the feature flag](https://docs.gitlab.com/ee/development/feature_flags/controls.html#cleaning-up) by running chatops command in `#production` channel
## Rollback Steps
- [ ] This feature can be disabled by running the following Chatops command:
```
/chatops run feature set --project=gitlab-org/gitlab feature_name false
```
issue