Draft: Allow charts module to target both sites

What does this MR do?

Related to #578 (closed)

Problem Statement

This MR is related to https://gitlab.com/gitlab-com/gl-infra/gitlab-dedicated/team/-/issues/1400 and is a component of the remediation of that issue.

At present, when GET is deploying Geo in a Hybrid Cloud Native environment, the Deployment process for the secondary sites is something like this: (see https://gitlab.com/gitlab-com/gl-infra/gitlab-dedicated/team/-/issues/1400#note_1191016463 for more detail)

• In playbooks/all.yml:
  • Step 1: In Ansible, generate the values for the secondary, but configured as a primary.
  • Step 2: Use Helm to Deploy Chart with Primary Values to Kubernetes Secondary
• In playbooks/gitlab_geo.yml:
  • Step 3: Use Ansible to fetches Helm values from Kubernetes Secondary (containing primary role values)
  • Step 4: Delete the Helm Deployment from the Kubernetes Secondary (!!)
  • Step 5: Using a regular expression onto the values YAML, replace role: 'primary' with role: secondary
  • Step 6: Use Helm to Deploy Chart with new adjusted value as a secondary

This process is repeated each time the configuration stage runs. This breaks Ansible's principal of idempotency and can leave the system in a broken state.

Solution

Ideally I would like:

1. Ansible deploys each cluster once, with the correct configuration the first time around
2. Follow-on deployments are idempotent. If nothing has changed in Ansible, the cluster is unchanged (and not torn down or misconfigured)
3. Ideally, run ansible-playbook across both sites simultaneously from a single ansible-playbook invocation. This is a more natural approach to using Ansible.

How can we go about doing this?

In order to do this, variables will need better scoping.

At present, GET advocates for placing variables in the all group. This is fine when all variables are identical, but in the case of multi-site deployments, there will be differences. Those site differences should be kept in per-site group variables rather than globally in all.

Unfortunately, when using hosts: localhost in a play, that play will only receive all scoped variables.

Instead, it's better to not rather use a specific host within a target site, and then use delegates_to: localhost. This has the advantage of running with the variables scoped to the host, not localhost. This allows tasks currently scoped to localhost to be run multiple times for different configurations. In the case of this MR, we can target the gitlab_charts tasks for both sites with the correct variables (and hence Geo roles) the first time the changes are pushed.

What about other GET users?

The goal is to make this change completely transparent to other GET users besides Dedicated.

Related issues

Author's checklist

When ready for review, the Author applies the workflowready for review label and mention @gl-quality/get-maintainers:

• Merge request:
  • Corresponding Issue raised and reviewed by the GET maintainers team.
  • Merge Request Title and Description are up-to-date, accurate, and descriptive
  • MR targeting the appropriate branch
  • MR has a green pipeline
  • MR has no new security alerts in the widget from the Secret Detection and IaC Scan (SAST) jobs.
• Code:
  • Check the area changed works as expected. Consider testing it in different environment sizes (1k,3k,10k,etc.).
  • Documentation created/updated in the same MR.
  • If this MR adds an optional configuration - check that all permutations continue to work.
  • For Terraform changes: set up a previous version environment, then run a terraform plan with your new changes and ensure nothing will be destroyed. If anything will be destroyed and this can't be avoided please add a comment to the current MR.
• Create any follow-up issue(s) to support the new feature across other supported cloud providers or advanced configurations. Create 1 issue for each provider/configuration. Contact the Quality Enablement team if unsure.

cc @ibaum