Auto parallelize as many parallel jobs as needed to keep tests under 5 minutes
Problem to solve
Splitting tests into multiple, parallel jobs is a great way to speed up the wall-clock time of a CI/CD pipeline. We have various issues about making this easier for people. It would be great to do this automatically for Auto DevOps as well. One problem is that with a single template, it's hard to know how far to parallelize. If we arbitrarily pick some parallelization factor, say 10 jobs, then for small test suites, it's overkill and each job takes seconds to run, and for large test suites, 10 jobs is just not enough. gitlab-ce
had ~70 parallel jobs last I checked.
Further details
(Include use cases, benefits, and/or goals)
Proposal
Let a developer specify how to split up parallel jobs while specifying a target wall-clock time rather than how many jobs. Let the system use historic JUnit data to know how long the test suite runs and split up the tests into as many jobs as necessary to hit the target timing.
What does success look like, and how can we measure that?
Enabling Auto DevOps for a large variety of projects would result in all projects having reasonable test times, regardless of test suite size, thus resulting in greater developer productivity and satisfaction with the product.