Next iteration of the CI/CD fair scheduling algorithm - capacity buckets
Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.
Description
Next iteration of fair scheduling.
Our current fair scheduling mechanism is not flexible and efficient enough. We need a better mechanism that will be more flexible and will not negatively impact our availability metrics.
Problem
The current fair scheduling algorithm is penalizing projects using shared runners that are creating a lot of builds. In such case the CI/CD processing gets delayed to ensure that projects that rarely see pipeline created in them take precedence in builds queuing.
The algorithm in its current form is built on top of an ORDER BY SQL query. This makes it perform slower can it could. It is a also very inflexible because all the changes to the algorithm itself require significant changes to the SQL query that is executed very frequently on GitLab.com (and a small change in the query plan can impact performance of the entire application).
Use Cases to support
- Queue pending jobs once users hit jobs limit up to new limit
- STRETCH - ability to enforce pipelines run in order (do not let jobs jump the line without a priority label)
Proposal
Build a new fair scheduling algorithm that performs better and is built on top of different concept, what would make it more flexible to extend, improve and iterate on.