[CI] Use vcluster for forward-looking smoke testing on Kubernetes 1.27, 1.28

What does this MR do?

Uses vcluster to create virtual clusters for testing review apps/environments.

Implements phase 1 of #5013 (closed).

This has some benefits:

• We can easily add new versions of Kubernetes to test against.
• The Kubernetes version of the host is irrelevant, so we can allow it to automatically update.
• We get much better isolation between review environments, as each branch gets its own virtual cluster.
• We have the ability to deploy different versions of CRDs.

Related issues

Related to gitlab-org&11331

Related to gitlab-org/distribution/team-tasks#1307 (closed)

Related to gitlab-org/distribution/team-tasks#1360 (closed)

Related to #5013 (closed)

Initial investigation and testing done in !3329 (closed).

Notes

At the moment, this MR takes the approach of using these vcluster review environments as simple tests against a specific Kubernetes version.

It does the following:

1. Installs a vcluster instance at the specified Kubernetes version
2. Installs the Helm Charts with a couple modifications
   • cert-manager is disabled
   • nginx-ingress Service is set to type NodePort to avoid creating a costly LoadBalancer
   • gitlab-runner is disabled since the domain is not configured
3. Ensures that the Webservice Deployment is rolled out successfully.
4. Destroys the environment automatically after 1 hour.

As noted in gitlab-org/distribution/team-tasks#1360 (closed), this is a helpful smoke test to ensure that we have valid manifests for the version we're deploying against, and the Webservice readiness probe gives us reasonable confidence that the application is running happily.

Deeper testing is done by QA jobs running against the traditional cloud provider instances, which do not change in this this MR.

Author checklist

See Definition of done.

For anything in this list which will not be completed, please provide a reason in the MR discussion.

Required

• Merge Request Title and Description are up to date, accurate, and descriptive
• MR targeting the appropriate branch
• MR has a green pipeline on GitLab.com
• When ready for review, follow the instructions in the "Reviewer Roulette" section of the Danger Bot MR comment, as per the Distribution experimental MR workflow

Expected (please provide an explanation if not completing)

• Test plan indicating conditions for success has been posted and passes - see MR pipelines
• Documentation created/updated
• Tests added/updated
• Integration tests added to GitLab QA
• Equivalent MR/issue for omnibus-gitlab opened
• Equivalent MR/issue for Gitlab Operator project opened (see Operator documentation on impact of Charts changes) - this change is related only to Charts CI, although based on the approach we take here we can investigate a similar workflow in Operator CI.
• Validate potential values for new configuration settings. Formats such as integer 10, duration 10s, URI scheme://user:passwd@host:port may require quotation or other special handling when rendered in a template and written to a configuration file.