Run new/updated E2E tests automatically even if quarantined

Problem

We intend to quarantine new E2E tests to verify that they're stable. However, quarantined tests need to be run manually in MRs. It would be very easy to forget to do so.

Proposal

Use change detection to automatically run new or updated tests, including those in quarantine.

It would be generally useful to have a job that automatically runs just the tests that were added/changed in an MR. There would be a few benefits:

  • We could focus on the results of the tests added/changed, without having to sort through all the other tests (though they should still run, in the existing jobs)
  • We could re-run those tests more efficiently, because if we retry the job it will only rerun the added/changed tests (in contrast with running every test in a scenario as we currently do).
  • If desired, we could turn off all other test jobs but still have coverage of the changes (This might be good to do if the only changes in an MR were to tests. No need to run tests that weren't changed. But we'd need to run the changed tests in all the scenarios that they will normally run in.)
  • We could automatically rerun the tests a few times (say, 10), to improve our chances of detecting flakiness
Edited by Mark Lapierre