[CI] Increase speed of feedback loop by making CI cluster tests manual and parallelizing jobs

Summary

Manual approval for CI cluster tests

Requires manual approval before proceeding to jobs that test against CI clusters. This reduces load on our clusters by allowing these tests to be intentional - not every commit warrants a full test in our CI clusters, and this helps us avoid false negatives when pipelines fail simply due to running out of CPU/memory in the clusters.

Parallelizing jobs

Along the same lines, this also parallelizes some jobs that don't touch the CI clusters to help quicken the feedback cycle.

Results

A full MR pipeline without this change takes about 40 minutes. With this change, a pipeline (without the jobs deploying to the cluster) will take ~10 minutes.

So the entire pipeline duration runs a bit faster thanks to the parallelized jobs, but the main benefit here is that multiple developers can push many times to the project and not overwhelm our cluster and inadvertently cause failures in CI due to limited resources.

This encourages more frequent and smaller commits, and speeds up the feedback loop, making our CI pipelines more helpful and intentional.

Considerations

This means we need to ensure that each MR contains at least one full QA pipeline before merging. To ensure we don't forget, we could add an MR description template with a checkbox for this item. That being said, this is probably the biggest reason to not move forward with this change - we could feasibly "forget" to do a full testing suite before merging.

Although, on that note, a pipeline that hasn't completed fully is denoted as blocked. Example: https://gitlab.com/gitlab-org/cloud-native/gitlab-operator/-/pipelines/362807548

That means that the overall MR wouldn't report that green check mark until the most recent pipeline has fully run and passed all jobs.

Closes #284 (closed)