Skip to content

Automatically run tests in e2e:package-and-test pipeline selectively in MR

Problem:

We currently do not run e2e:package-and-test-* suite automatically and selectively on every commit on every MR due to non availability of runner resources. This issue will be solved with gitlab-org/gitlab-qa#261 (closed) and https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/14333.

Until the issues regarding non availability of runner resources is resolved, we would want to only run the e2e:package-and-test-* pipeline once before merging the MR.

Although the smoke and reliable/blocking e2e tests do run automatically against review-apps and provide some coverage but they do not include orchestrated scenarios. This results in missing coverage and regression being leaked into master. Here is a recent example of this: gitlab-org/gitlab#373803 (closed) => This is not true anymore. See this comment for details.

Proposal

1st iteration - COMPLETE

  1. Add a comment/note that prompts the developer to run the e2e:package-and-test-ee pipeline (which selectively executes the test) in the MR.
  2. This comment will need to be marked with ✅ to acknowledge the message and indicate that the pipeline was run.

2nd iteration

  1. Make the e2e:package-and-test-ee job run automatically when selective tests are detected. To keep the load on runners under control, this can be rolled out by initially running the job automatically once an MR has been approved (i.e., it has pipeline:mr-approved label)

Roll out for:

  1. Replace the message for manually running trigger-omnibus-and-follow-up-e2e with a message indicating that the jobs have started.

  2. Switch to using trigger-omnibus-and-follow-up-e2e for these jobs and make the job fail but not block the pipeline. (See)

  3. Full roll-out by allowing running the selective tests on every push/rebase if the load can be taken by the runners and costs are under control.

3rd iteration

The 3rd iteration has its own issue: #1804

  1. If there are NEW failures in the e2e test, create a resolvable comment on the MR.
  2. The comment will unresolved on each pipeline run.

Prerequisites:

  • Make the selective jobs run tests in parallel => MR

History:

We had some efforts performed earlier in regards to running the package-and-test selectively on MRs:

  1. Use MR labels to selectively run package-and-qa... (gitlab-org/gitlab!89511 - merged)
  2. The above MR hit a road block: package-and-qa job does not auto run based on M... (gitlab-org/gitlab#366871 - closed)
  3. The road block was resolved with: Integrate orchestrated e2e tests in to gitlab p... (gitlab-org/gitlab!92039 - merged)
  4. But we still decided to not move ahead with enabling auto running of package-and-test selectively on MRs due to limited runner resources available to the job.

Related

Edited by Sanad Liaquat