Run Reliable QA Tests automatically as part of MR pipelines

Currently we run Smoke QA tests as part of MR pipelines with full runs being available optionally. This is to maintain a balance of coverage and cost (in time, money, etc...).

However, as some recent incidents have shown (gitlab-com/gl-infra/production#5712 (closed), gitlab-com/gl-infra/production#5748 (closed), gitlab-com/gl-infra/production#5756 (closed)) bugs are making their way arguably too far down the deployment chain, which requires manual intervention.

The above incidents would've been prevented if Reliable tests were running automatically in MRs. Task then is to explore if it's viable to run Reliable tests on every MR in terms of time and costs.

The proposal is to utilise the existing Review Apps functionality by adding a dedicated job for reliable tests:

• Add review-qa-reliable job that runs against Review Apps and is allowed to fail (MR: gitlab-org/gitlab!74147 (merged))
• Add daily slack reporting for any failures to monitor it closely (MR: gitlab-org/quality/toolbox!88 (merged), Pipeline Schedule: https://gitlab.com/gitlab-org/quality/toolbox/-/pipeline_schedules/183923/edit, Slack Channel: #review-qa-failures)

V2

• Make review-qa-reliable job mandatory (not allowed to fail) for MRs with Frontend and QA changes once stable (MR: gitlab-org/gitlab!76802 (merged))
• Run both review-qa-smoke and review-qa-reliable jobs in Backend MRs automatically
  • Automatically deploy Review App for changes in app/controllers (MR: gitlab-org/gitlab!77897 (merged))

FY23 Q1 (Carried forward)

• Automatically deploy Review App for changes in app/models (MR: gitlab-org/gitlab!80121 (merged))
• Automatically deploy Review App for changes in lib/gitlab (MR: gitlab-org/gitlab!84236 (merged))