FY22 Q1 KR: Reduce setup time for Merge Request regression tests by 50% from 40 to 20mins => 65%
The following needs to be done to achieve this KR:
-
Analyse whether review-apps can be used for end-to-end tests. YES - gitlab-org/gitlab!55583 (merged) -
Fix tests failing only on Review Apps => 100% complete
- gitlab-org/gitlab#30448 (closed) -
Figure out why review-qa-smoke
tests still sometimes fail=> 50%
(gitlab-org/gitlab#323741 (closed)) -
Analyse the gap between using review-qa-all
andpackage-and-qa
- How can this be addressed optimally? -
Speed up package-and-qa
- gitlab-org/omnibus-gitlab#5429 (closed) -
Make it possible to skip tests that can't be run on Review Apps - https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/821 => 100%
-
Test running Review Apps as a hybrid architecture with Gitaly off k8s in progress, 10%
-
Make review-qa-all
mandatory forqa
folder specific MRsin progress, 80%
-
Set up monitoring/alerting so that we don't have unexpected failures due to resource constraints in Review Apps - https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/834 in progress, 80%
-
Document the new pipeline/job related details and communicate the same. -
Measure the pipeline/job success regularly.
** Stretch Goals:
- Can the
security fixed
run this as well?
Monitoring
Edited by Ramya Authappan