Merge Train next iteration: run tests in Merge train pipelines
Now that we're using the Merge Train mechanism, we should think about the testing strategy we want to use in Merge Train pipelines.
Proposal
| Proposal | Pros | Cons |
|---|---|---|
Run a predictive pipeline + pre-merge-checks job |
That would ensure we're testing the most relevant tests against a merge train commit, which prevents two MRs touching the same codebase from introducing conflicting changes. By keeping the pre-merge-checks job, we would have the same "quality checks" we have right now, plus the assurance that we catch most of the conflicting changes. |
Predictive tests can have test gaps and that would make Merge Train pipelines slower (P50: 17m; P80: 28m, based on https://app.snowflake.com/ys68254/gitlab/#/ep-predictive-pipelines-dUprrnolY). |
| Run a full pipeline | Best solution to keep master green! |
Full pipelines are slow and that would lead to long merge trains. Flaky tests could make the train derails, and lead to "Merge Train traffic jam". |
Edited by Peter Leitzen