Actively expand GDK test suite independent of reliable tag
Using reliable
tag is problematic because tests tend to be much less stable on live
environments. This limits ability to shift left and start blocking mrs via tests that tend to be more stable.
With gitlab-org/gitlab!141802 (merged) merge we introduced temporary blocking
tag that can be added to non-orchestrated tests which allows them to be executed in blocking job. It also now runs full suite on master
runs to start collecting metrics on gdk test stability for rest of the test suite.
Once all of non-orchestrated, non staging only tests are converted to blocking
and flaky/failing tests are quarantined, we can retire blocking
tag and run full suite by default always.
Metrics on gdk test failures and retries: https://dashboards.quality.gitlab.net/d/Csi6nbvnk/top-failures?orgId=1&refresh=30s&var-run_type=e2e-test-on-gdk