Allow Crystalball to use described strategy while also writing data for coverage strategy
What and why
We don't really want to run all the tests twice with each strategies, so it'll be nice if there's a way to let Crystalball uses one strategy to run and also use another strategy for just writing data. This can be found at spec/crystalball_env.rb
This will greatly help with #466068 so that we know better which strategy can work better. We need to know that:
- Is described strategy more accurate?
- Is coverage strategy more accurate?
- Is using both with #455826 more accurate?
Background
After !147076 (merged) we changed the Crystalball strategy from coverage to described.
However we can't really tell if it's doing better or not. Chart for the transition rate from predictive to non-predictive isn't fully reliable due to flaky tests and other unrelated failures as well (see gitlab-org/quality/engineering-productivity/team#495). Back then we want to do this because we want to avoid running too many tests in the predictive pipelines, because coverage can sometimes give a long list of related spec files which are too many.
Since then we implemented quite a lot of limiting methods to avoid hitting into the worst case running a ton of spec files, and we can also implement #455826 which can potentially improve the prediction accuracy while reducing the number of spec files we run.
The thing is, now we don't really have data to compare with because the current data we have is only for the described strategy: https://gitlab-org.gitlab.io/gitlab/crystalball/packed-mapping.json.gz
This is because Crystalball can only use one strategy at a time, and write data with that strategy.
Implementation note
-
spec/crystalball_env.rb
for how we're generating the mapping file -
tooling/lib/tooling/find_tests.rb
for how we're using the mapping file to find the tests - Crystalball goes over the strategies to update internal data, and then at the end it writes to a file. It needs to run the specs to update and generate data
- We're using
scripts/generate-test-mapping
to merge all the YAML files Crystalball writes for each CI job