Automatically run tests in e2e:package-and-test pipeline selectively in MR
Problem:
We currently do not run e2e:package-and-test-*
suite automatically and selectively on every commit on every MR due to non availability of runner resources. This issue will be solved with gitlab-org/gitlab-qa#261 (closed) and https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/14333.
Until the issues regarding non availability of runner resources is resolved, we would want to only run the e2e:package-and-test-*
pipeline once before merging the MR.
Although the smoke and reliable/blocking e2e tests do run automatically against review-apps and provide some coverage but they do not include orchestrated scenarios. This results in missing coverage and regression being leaked into master. Here is a recent example of this: gitlab-org/gitlab#373803 (closed) => This is not true anymore. See this comment for details.
Proposal
1st iteration - COMPLETE
- Add a comment/note that prompts the developer to run the
e2e:package-and-test-ee
pipeline (which selectively executes the test) in the MR. - This comment will need to be marked with
✅ to acknowledge the message and indicate that the pipeline was run.
2nd iteration
- Make the
e2e:package-and-test-ee
job run automatically when selective tests are detected. To keep the load on runners under control, this can be rolled out by initially running the job automatically once an MR has been approved (i.e., it has pipeline:mr-approved label)
Roll out for:
-
devopscreate => gitlab-org/gitlab!133825 (merged) -
devopsgovern => gitlab-org/gitlab!135217 (merged) -
devopsmanage => gitlab-org/gitlab!135217 (merged) -
devopsplan => gitlab-org/gitlab!138146 (merged) -
devopsverify => gitlab-org/gitlab!138146 (merged) -
devopspackage => gitlab-org/gitlab!138146 (merged) -
devopsmonitor => gitlab-org/gitlab!142786 (merged) -
devopsdata stores => gitlab-org/gitlab!142786 (merged) -
devopsfulfillment => gitlab-org/gitlab!142786 (merged) -
devopssystems => gitlab-org/gitlab!142786 (merged) -
devopssecure -
devopsmodelops -
devopsgrowth -
Remove the code for selecting specific stages and rollout to all
-
Replace the message for manually running
trigger-omnibus-and-follow-up-e2e
with a message indicating that the jobs have started. -
Switch to using
trigger-omnibus-and-follow-up-e2e
for these jobs and make the job fail but not block the pipeline. (See) -
Full roll-out by allowing running the selective tests on every push/rebase if the load can be taken by the runners and costs are under control.
3rd iteration
The 3rd iteration has its own issue: #1804
- If there are NEW failures in the e2e test, create a resolvable comment on the MR.
- The comment will unresolved on each pipeline run.
Prerequisites:
-
Make the selective jobs run tests in parallel => MR
History:
We had some efforts performed earlier in regards to running the package-and-test
selectively on MRs:
- Use MR labels to selectively run package-and-qa... (gitlab-org/gitlab!89511 - merged)
- The above MR hit a road block: package-and-qa job does not auto run based on M... (gitlab-org/gitlab#366871 - closed)
- The road block was resolved with: Integrate orchestrated e2e tests in to gitlab p... (gitlab-org/gitlab!92039 - merged)
- But we still decided to not move ahead with enabling auto running of
package-and-test
selectively on MRs due to limited runner resources available to the job.
Related
-
@acunskis
Already has an MR in review to enable selective test execution on review-apps - Earlier MR to enable selective test execution on
package-and-test
based on MR labels