Parallelize `rspec foss-impact` by test level based on Knapsack data
What does this MR do and why?
Prerequisite of https://gitlab.com/gitlab-com/gitlab-OKRs/-/work_items/122796732.
- This generalize the RSpec pipeline generation script and port it to Ruby, so that later we can reuse the script for https://gitlab.com/gitlab-com/gitlab-OKRs/-/work_items/122796732.
- This also split the
rspec foss-impactjobs by test level and parallelize based on the latest Knaspack report (it calculates the average duration of test files per test level).
You can try runnnig the script locally:
$ curl -o matching_tests-foss.txt https://gitlab-org.gitlab.io/-/gitlab/-/jobs/3794959154/artifacts/rspec/matching_tests-foss.txt
$ curl -o knapsack-report-master.json https://gitlab-org.gitlab.io/gitlab/knapsack/report-master.json
$ scripts/generate_rspec_pipeline.rb -f "matching_tests-foss.txt" -t ".gitlab/ci/rails/rspec-foss-impact.gitlab-ci.yml.erb" -k knapsack-report-master.json
Optimal node count for 0 migration RSpec files is 0.
Optimal node count for 0 background_migration RSpec files is 0.
Optimal node count for 69 unit RSpec files is 2.
Optimal node count for 8 integration RSpec files is 1.
Optimal node count for 10 system RSpec files is 2.
# and then inspect the generated pipeline file:
$ cat .gitlab/ci/rails/rspec-foss-impact.gitlab-ci.yml.erb.yml
Example of a pipeline generated with dynamic parallelization: https://gitlab.com/gitlab-org/gitlab/-/pipelines/781962519
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.
Edited by David Dieulivol