Skip to content

Improve predictive jobs parallelization

What does this MR do and why?

Instead of using the average test file duration for a given test level type, we now use the duration of the actual test files that will run in the predictive jobs. That way, if we only run very slow test files, the computation will take it into account and parallelize on more jobs.

Screenshots or screen recordings

Testing the commit from !132937 (merged) with !133195 (closed).

Before After
Screenshot_2023-10-03_at_12.40.57 Screenshot_2023-10-03_at_12.48.09

How to set up and validate locally

Without the optimization we have the following parallelization:

[GenerateRspecPipeline] Optimal node count for 19 unit RSpec files is 1.
[GenerateRspecPipeline] Optimal node count for 3 integration RSpec files is 1.
[GenerateRspecPipeline] Optimal node count for 4 system RSpec files is 1.

With the optimization, we'd get the following (note the different parallelization difference for unit test files!):

[GenerateRspecPipeline] Optimal node count for 19 unit RSpec files is 16.
[GenerateRspecPipeline] Optimal node count for 3 integration RSpec files is 3.
[GenerateRspecPipeline] Optimal node count for 4 system RSpec files is 3.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Rémy Coutable

Merge request reports