Fix automated Geo installation QA tests

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

• Work on this issue

Problem

We used to run scheduled pipelines which install a 1k+1k environment and a 3k+3k environment, and run basic Geo QA tests against them.

https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/tenant-scale/geo/scheduled_pipelines/

This is not currently working. This would have caught at least one simple, severe bug which we instead learned about by chance in staging-ref. By the time the fix was merged, the bug was released to customers, and they had to wait for backports to be released.

Proposal

This must be fixed (and more QA tests must be added in other issues) in order to meet our standard of reliability.

References

• https://gitlab.com/gitlab-org/geo-team/geo-ci/
• We started fixing it but there were many problems to get through. For a taste, see threads in omnibus-gitlab!8634 (closed)