Skip to content

If an MR changes only e2e tests, run only those e2e tests

Sanad Liaquat requested to merge qa-shl-selective-test-execution-first-iteration into master

What does this MR do and why?

Adds and uses a script tooling/bin/qa/check_if_qa_only_spec_changes that takes changed files as input and creates a new file with either end-to-end test files (with leading qa trimmed) or an empty file. It also sets QA_TESTS variable with the contents of the new file and sends this variable to the triggered Omnibus pipeline.

Once this MR is merged, we will need to pass the QA_TESTS variable from the Omnibus to GitLab-QA. GitLab-QA is already set to use QA_TESTS to run only those tests.

Related issue: https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/564

Screenshots or screen recordings

These are strongly recommended to assist reviewers and reduce the time to merge your change.

How to set up and validate locally

This is tricky to test in the current MR since the check_if_qa_only_spec_changes script only writes to file if the only only changes in the MR are e2e tests but this MR changes other files.

The tooling/bin/qa/check_if_qa_only_spec_changes script can be tested locally by running tooling/bin/qa/check_if_qa_only_spec_changes /path/to/changed_files.txt /path/to/qa_changed_files.txt. Following testcases can be tested:

  1. When changed_files.txt file contains only e2e tests such as: qa/qa/specs/features/ee/browser_ui/fulfillment/purchase/free_trial_spec.rb qa/qa/specs/features/ee/browser_ui/fulfillment/purchase/purchase_ci_spec.rb. This with produce a qa_changed_files.txt with: qa/specs/features/ee/browser_ui/fulfillment/purchase/free_trial_spec.rb qa/specs/features/ee/browser_ui/fulfillment/purchase/purchase_ci_spec.rb

  2. When changed_files.txt file contains e2e tests as well as non-e2e tests such as qa/qa/specs/features/ee/browser_ui/fulfillment/purchase/upgrade_group_spec.rb app/assets/javascripts/api/analytics_api.js. This will produce an empty qa_changed_files.txt file.

  3. When changed_files.txt file contains only non-e2e tests such as qa/qa.rb app/assets/javascripts/api/analytics_api.js. This will produce an empty qa_changed_files.txt file.

To test if QA_TESTS variable is being set, I commented some code in check_if_qa_only_spec_changes so that it writes all changed files to a file. The QA_TESTS was set with all changes: https://gitlab.com/gitlab-org/gitlab/-/jobs/1832378335#L51

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Sanad Liaquat

Merge request reports