Add initial support for Hybrid Reference Architectures with Helm (GCP)

Relates https://gitlab.com/gitlab-org/quality/gitlab-environment-toolkit/-/issues/79 https://gitlab.com/gitlab-org/quality/gitlab-environment-toolkit/-/issues/77 gitlab-org/quality/reference-architectures#41 (closed)

Closes https://gitlab.com/gitlab-org/quality/gitlab-environment-toolkit/-/issues/90

MR adds in initial support to deploy Cloud Native Hybrid Reference Architectures via Helm and our official Charts to the GitLab Environment Tookit!

Note that this functionality is opinionated by default. GET is specifically gaining the ability here to deploy what's officially supported in terms of our Cloud Native Hybrid Reference Architectures. This is namely GitLab Webservice, Sidekiq and other supporting deployments such as Ingress, etc.... on Kubernetes via our Helm Charts along with hooking these up to Omnibus built backends. Support for more components via Helm will be determined on if we recommend other GitLab components be moved to Kubernetes in the Cloud Native Hybrid Reference Architectures. At the time of writing no immediate changes are planned there.

With that note said on to the exciting parts:

New Features

• Provisioning Kubernetes Cluster, Node Pools and associated network changes on GCP is now automated in Terraform via our GCP Ref Arch Module
  • Cluster and associated network requirements are intelligently only configured when Node Pools are requested.
  • Object Storage provisioning on GCP now supports either a consolidated bucket or separate ones (recommended). This will be a breaking change in the future where separate buckets become the default, likely for the main v1.0.0 release, but for now the consolidated bucket will stay as the default to allow a grace period.
  • Provisioning for other cloud providers will be explored in subsequent MRs.
• Configuring a Helm Chart deployment is now automated in Ansible on GCP with the following features:
  • Automatic configuration of the GitLab Chart to match our recommended hybrid architectures. Ansible will automatically calculate things such as replica sizing based on Node Pools sizes as well as hook up the chart to Omnibus based backends.
  • Automatic configuration of required secrets including solutions for syncing required GitLab secrets between the Chart and Omnibus nodes.

Updates

• Detecting the Postgres Primary node for Migrations via the gitlab-ctl get-postgresql-primary command is now offloaded to the first Postgres node instead of running on GitLab Rails. Through this change both either Rails or the Charts can find the primary and set up migrations accordingly.
• Post Configuration steps such as configuring settings that can only be done via API, etc... are now performed on the Ansible node (localhost) and have been updated to generate the require Access Token on either a GitLab Rails node of the GitLab Charts Task Runner pod.

Notes

• With this update kubectl and helm will obviously be requirements for Ansible configuring Hybrid environments.
• Due to the way Kubernetes authenticates separately it's doesn't lend itself well to automation, especially for Ansible with the provisioned cluster (Terraform themselves call it a challenge). There's several ways a user can configure the .kubeconfig file required for automation. As a convenience though GET will attempt to run the recommended way to do this for each cloud with Ansible (e.g. gcloud container clusters get-credentials for GCP) but this can also be disabled by the user if preferred. For when this isn't appropriate this can also be disabled by the user if preferred and as long as the kubeconfig file is correctly configured to pointing to the cluster (and the right context selected) GET will support that as well.
• As noted above a future breaking change will be announced in this release that Object Storage buckets will switch to be separated as recommended for all installs although consolidated buckets will remain the default for now.

WIP

A few things are still being worked on for this MR:

• Automated webservice Puma tuning for optimum performance (review ongoing in separate issue)
• A new streamlined way to set GitLab install version across both Omnibus and Charts.
• Lay groundwork for handling multiple Object Storage buckets correctly in Rails. To be kept as single bucket for now for backwards compatability.