FY22 Q1 KR: Reduce setup time for Merge Request regression tests by 50% from 40 to 20mins => 65%

The following needs to be done to achieve this KR:

• Analyse whether review-apps can be used for end-to-end tests. YES - gitlab-org/gitlab!55583 (merged)
• Fix tests failing only on Review Apps => 100% complete - gitlab-org/gitlab#30448 (closed)
• Figure out why review-qa-smoke tests still sometimes fail => 50% (gitlab-org/gitlab#323741 (closed))
• Analyse the gap between using review-qa-all and package-and-qa - How can this be addressed optimally?
• Speed up package-and-qa - gitlab-org/omnibus-gitlab#5429 (closed)
• Make it possible to skip tests that can't be run on Review Apps - https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/821 => 100%
• Test running Review Apps as a hybrid architecture with Gitaly off k8s in progress, 10%
• Make review-qa-all mandatory for qa folder specific MRs in progress, 80%
• Set up monitoring/alerting so that we don't have unexpected failures due to resource constraints in Review Apps - https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/834 in progress, 80%
• Document the new pipeline/job related details and communicate the same.
• Measure the pipeline/job success regularly.

** Stretch Goals:

• Can the security fixed run this as well?

Monitoring

• review-deploy duration
• review-build-cng duration
• review-qa-smoke duration | success rate
• review-qa-all duration
• package-and-qa duration
• GCP usage dashboard | alerts