Create a multi-node secondary on staging with GET

Summary

In https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/12517 and recent working group meetings, we discussed the possibility of using the GitLab Environment Toolkit (GET) as an option to set-up a multi-node Geo secondary on staging. It is understood that this is an initial iteration and may be a temporary solution until we settle upon the appropriate tooling.

With https://gitlab.com/gitlab-com/gl-infra/mstaff/-/issues/34#note_471435886, we've decided to start with a 3k reference architecture.

With this issue, we'll plan and execute the creation of a multi-node secondary site on staging.

Steps

A broad outline of the steps to create the multi-node secondary. We'll continue to expand with more detail or consider promoting this to an epic and adding issues for each step.

• Set up necessary accounts and shared VM to run GET
• Using GET, deploy a 3k reference architecture to a yet-to-be determined GCP project
• Set up database replication using a Patroni standby cluster on the secondary site
• Configure the secondary node
• Add the secondary node to the primary
• Set up selective sync
• Document the required steps and automate
• Set up monitoring

We can also look to the epic for the initial set up of Geo on staging for some guidance.