Skip to content

Reduce the number of backend test predictions

David Dieulivol requested to merge 450374-prediction_limit_crystalball into master

From Draft to Ready

Context

Closes #450374 (closed).

Implements the feature introduced in gitlab-org/ruby/gems/test_file_finder!46 (merged).

What does this MR do and why?

Reduce the number of backend test predictions for the topmost 5% of mappings only.

This is not as important for the DescribedClass Crystalball strategy as it would have been for the Coverage Crystalball strategy, but I think it's still useful to have it working for the top 5% cases, and in to have this mechanism available in general if we want to use other crystalball strategies, or the same strategies with different thresholds for making pipelines tiers 👍

Proof of work

Change a file that will trigger 36 backend test files

  • Expected: We will reduce the number of backend test files to run by 50%.
  • Actual: ✅ 
$ cat rspec/changed_files.txt
app/models/concerns/spammable.rb

$ cat crystalball/mapping.json | jq '."app/models/concerns/spammable.rb" | length'
36

# Clean the previous results
rm -rf rspec/matching_tests*

# Run https://gitlab.com/gitlab-org/quality/engineering-productivity/team/-/blob/main/runbooks/predictive-test-selection.md#generate-test-predictions-like-detect-tests-would locally

$ cat rspec/matching_tests.txt | wc -w
18

Change a file that will trigger 10 backend test files

  • Expected: We will keep the number of backend test files to 10.
  • Actual: ✅ 
$ cat rspec/changed_files.txt
app/finders/issuable_finder.rb

$ cat crystalball/mapping.json | jq '."app/finders/issuable_finder.rb" | length'
10

# Clean the previous results
rm -rf rspec/matching_tests*

# Run https://gitlab.com/gitlab-org/quality/engineering-productivity/team/-/blob/main/runbooks/predictive-test-selection.md#generate-test-predictions-like-detect-tests-would locally

$ cat rspec/matching_tests.txt | wc -w
10
Edited by David Dieulivol

Merge request reports