Tests for clusters / Kubernetes / Auto DevOps stories

Currently, we manually test MRs which are related to Cluster story. e.g. Configure gitlab.yml for Google OAuth setting -> Spin up GDK -> Input cluster params in a form -> Wait 5 minutes until it's done (Because it takes 5 min to create a cluster on GKE) -> Test around -> Install app -> etc

This is painful that it's required repetitive actions.

We should have a complete test flow to walkthrough all features.

e.g.

Begin

1. Launch Omnibus GitLab
2. Create a project
3. Create a GKE cluster (It's created on our dev GCP project)
4. Install apps (Ingress/Runner/Prometheus)
5. Enable Auto DevOps
6. Import sample ruby project (which designed for Auto DevOps)
7. Run pipeline. Pipeline: auto build -> auto test -> deploy to the cluster. This should be executed by the installed runner.
8. Check the pipeline status. Check the access to the deployment. If it's ok, QA passed.

Ensure

1. Dispose the GKE cluster

/cc @fabio @grzesiek @ayufan @nolith @matteeyah

Exploration Notes

• Exposed port + shell runner on exposed server idea may work but put to the side as this won't be easy to run locally and in trying to test this locally I came to the realisation ngrok solves our problem and works locally
• While working with ngrok I realised that the licensing cost is too high + the way they license their product doesn't really fit our use-case. For example even with the Pro plan in theory we can only really run 1 QA spec (as we require 2 tunnels 1 for gitlab, 1 for registry) and then it still didn't work because of the 20 connections/60 second limit. This was exceeded by docker push pushing many layers to the registry.
• Using locally running gitlab runner was tricky because of Docker in Docker since auto devops requires this to build images etc. so we decided to go with runner on K8s cluster. Additionally this is beneficial since it tests this deploy runner to K8s integration
• Running external runner on K8s means gitlab as well as registry will need to be internet exposed
• I tested ngrok and localtunnel for tunneling. Since localtunnel was open source and still maintained it seemed preferable, however I had some trouble getting the server component running and the publicly available localtunnel.me server has a file upload limit that is exceeded by our docker image uploads. The problem I had with getting the server running on a subdomain seemed to concern me as the maintainer had minimal docs and gave short and generally difficult to understand answers on issues.
• After struggling with localtunnel server I switched to ngrok version 1 (which is open source) and set up a server for this. The server works and does not have any problems with docker uploads