Improve predictive jobs parallelization
What does this MR do and why?
Instead of using the average test file duration for a given test level type, we now use the duration of the actual test files that will run in the predictive jobs. That way, if we only run very slow test files, the computation will take it into account and parallelize on more jobs.
Screenshots or screen recordings
Testing the commit from !132937 (merged) with !133195 (closed).
Before | After |
---|---|
How to set up and validate locally
Without the optimization we have the following parallelization:
[GenerateRspecPipeline] Optimal node count for 19 unit RSpec files is 1.
[GenerateRspecPipeline] Optimal node count for 3 integration RSpec files is 1.
[GenerateRspecPipeline] Optimal node count for 4 system RSpec files is 1.
With the optimization, we'd get the following (note the different parallelization difference for unit
test files!):
[GenerateRspecPipeline] Optimal node count for 19 unit RSpec files is 16.
[GenerateRspecPipeline] Optimal node count for 3 integration RSpec files is 3.
[GenerateRspecPipeline] Optimal node count for 4 system RSpec files is 3.
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.
Edited by Rémy Coutable