Merge Train next iteration: run tests in Merge train pipelines
Now that we're using the Merge Train mechanism, we should think about the testing strategy we want to use in Merge Train pipelines.
Proposal
Proposal | Pros | Cons |
---|---|---|
Run a predictive pipeline + pre-merge-checks job |
That would ensure we're testing the most relevant tests against a merge train commit, which prevents two MRs touching the same codebase from introducing conflicting changes. By keeping the pre-merge-checks job, we would have the same "quality checks" we have right now, plus the assurance that we catch most of the conflicting changes. |
Predictive tests can have test gaps and that would make Merge Train pipelines slower (P50: 17m; P80: 28m, based on https://app.snowflake.com/ys68254/gitlab/#/ep-predictive-pipelines-dUprrnolY). |
Run a full pipeline | Best solution to keep master green! |
Full pipelines are slow and that would lead to long merge trains. Flaky tests could make the train derails, and lead to "Merge Train traffic jam". |
Edited by Peter Leitzen