Determine how to build out a multi-node secondary site

Currently we are using a single node for Geo as our secondary-site in our staging environment. This is not sufficient for the staging environment as we are not syncing all available data, and the workload on staging is too high for us to perform a full cycle failover event. We need to determine how we want to build out a larger secondary-site in order to accomplish this goal. But we first must know HOW to build out this infrastructure.

Use existing Tooling

  • This assumes the use of our existing chef infrastructure
  • Uses our existing terraform capabilities

We'd probably utilize the above to spin up a larger multi-node site in a differing region in our staging GCP project.

Upsides

  • Known to all team members that managed our infrastructure
  • ...

Downsides

GET

GitLab Environment Toolkit: https://gitlab.com/gitlab-org/quality/gitlab-environment-toolkit

We'd probably utilize this to stand up something temporary mayhaps? We'd then need to connect this to our existing staging environment, perform a sync. Then we can perform the necessary testing.

Upsides

  • Currently heavily utilized by QA
  • Better aligns the configuration of our infrastructure with reference architectures
  • ...

Downsides

  • This tool is currently unfamiliar with all of the infrastructure team
  • This tool provides a completely different mechanism to manage our infrastructure
  • ...

Utilize this issue to discuss the pros and cons of each configuration. We'll spin up issues after a decision is made to cover the implementation details.

Edited by John Skarbek