Prioritize previously failed tests
Description
When CI fails, it's common to push changes to see if the tests now pass, but still have them fail, then keep looping until resolved. Each time, you have to wait for the entire CI/CD pipeline to run before seeing results.
One improvement is being able to test on local changes so you don't have to push commits to GitLab (and don't pollute the commit log) (https://gitlab.com/gitlab-org/gitlab-ce/issues/19267).
On top of that, it would be smart to prioritize jobs/tests that previously failed. e.g. if 19 jobs succeed and 1 fails, and they're not all run in parallel (due to resource constraints, then start with the most suspect test first.
Once we understand tests and not just jobs, it would be even better collect all the failed tests from within multiple jobs, and put them into a single (or multiple) prioritized job.
I imagine being able to create a fast-fail job that does only previously failed jobs, and if they still fail, then don't even bother going to the next stage to run the rest of the tests. But if this is the first run on a particular ref, or no previous failures were detected, then just skip the fast-fail job and run the entire test suite.
Alternatively, if you have enough available resources, you could start running some of the full test suite, but prioritize failed tests so at least all of them are picked up before non-prioritized tests.
Lastly, when combined with a CLI to run tests, how about an option to only test previously failed tests. Sometimes you're confident you didn't introduce anything new, and just want a tight loop of checking the failed test again. When ready, you'll push back to origin, so the full suite will run again anyway.
Proposal
fast-fail:
stage: fast-fail
script: rspec $PREVIOUS_FAILED_TESTS
when: previous-failures
full-test:
stage: test
script: rspec $PREVIOUS_SUCCESSFUL_TESTS
Links / references
Documentation blurb
Overview
What is it? Why should someone use this feature? What is the underlying (business) problem? How do you use this feature?
Use cases
Who is this for? Provide one or more use cases.
Feature checklist
Make sure these are completed before closing the issue, with a link to the relevant commit.
-
Feature assurance -
Documentation -
Added to features.yml