Create developer tooling to be able to run tests that failed in a pipeline locally
Follow up from #323151 (closed) to experiment with a tooling to run tests that failed in a pipeline locally
User story: As a developer, I can run tests that failed in my latest CI pipeline locally, so that I can be sure I have fixed the tests and the next pipeline would be more likely to be green.
Rationale
Related discussion on running failing rspec tests from the previous pipeline runs first: https://docs.google.com/document/d/1yzHuosvykzb_kaldjGP9I6wxpNpMXdNEvEFBkHyZpXs/edit#bookmark=id.5daxano9kcnq. Instead of running failing rspec tests from the previous pipeline in the next pipeline, we could also make it easier to run these failed tests locally. This would eliminate the need to wait for ~10 minutes for setup-test-env in CI as well as the overhead of pushing.
metric | changes |
---|---|
MR failed pipeline duration | no significant change as pipeline does not change |
Average cost per merge request pipeline | no significant change as pipeline does not change |
Number of pipelines per MR | may be reduced, as less back & forth to commit, push, wait for CI, fix regressions, repeat |
Number of cancelled pipelines per MR | may be reduced, as less back & forth to commit, push, wait for CI, fix regressions, repeat |
Time to review to measure feedback effectiveness | low correlation - possibly less, with less back and forth above and thus less context switching, MR can be made ready for review quicker |
Narrow MR Rates as a measure of productivity | low correlation - possibly increase if time to review/merge is shorter |
How much time has elapsed between pipelines | not enough correlation |