Optimize CI job scheduling within stage

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

Description

Generally the execution time of a CI stage is defined by the slowest job.

Example:

We have a few jobs that uses qemu emulation and some that don't. The run time of the qemu build is 45min vs. 5min for native code.

Our build server is able to handle 12 concurrent builds and we have more than 12 jobs in the stage. By clever naming we schedule the four qemu builds first and while the native builds share the remaining 8 cores. In this way the total execution time is roughly 45 minutes.

This scheduling strategy can be generally applied.

Proposal

Considering that Gitlab CI has access to the execution time of all previous jobs, it is possible estimate the execution time for the current run (average of last 10).

Using this information it is possible to find an optimal scheduling.

Example: 1x30min jobs + 6x5min jobs with concurrency 2. This can either take 30 min total or in the worst case 45min.

Overview

  • What is it?

Proposal for CI job scheduling algorithm.

  • Why should someone use this feature?

Faster (overall build time). More efficient use of builders and hence less need for auto scaling.

  • What is the underlying (business) problem?

Non-optimal CI scheduling.

  • How do you use this feature?

Currently we manually optimize scheduling. However, an automatic solution would be superior.

Use cases

Who is this for? Provide one or more use cases.

Companies with fixed number of concurrent runners can optimize their usage.

Companies with autoscaling can get by with fewer runners.

Feature checklist

Make sure these are completed before closing the issue, with a link to the relevant commit.

Edited by 🤖 GitLab Bot 🤖