Selective test execution: If an MR only changes end-to-end tests, run only those end-to-end tests
Problem
When we submit an MR that only makes changes to a test, we still run a full pipeline including all the test suites.
This is potentially more of a concern for end-to-end tests because package-and-qa
takes hours to run, but it's also relevant to all other tests.
Completed Tasks (FY22 Q4)
- Fix an issue with jobs that use the
:parallel
keyword where theQA_TESTS
variable is not honoured => GitLab-QA MR - Run only the end-to-end tests that changed in an MR if only changes in the MR are the end-to-end tests. => GitLab MR, Omnibus MR
- Add ability to fetch no of examples in a Test Scenario without running the tests => MR
Pending Tasks (FY23 Q1)
-
Use the mechanism introduced in previous iteration to generate files for each pipeline that will run one or more tests. => MR -
Do not run the jobs that have zero examples. One idea is to use a child pipeline based on the changed files as described at https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/564#note_394535673 => MR
Edited by Sanad Liaquat