Skip to content

Named queues in GitLab CI

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

Description

Can improve GitLab CI queuing system to be more scalable by moving away from tags, in favor of named queues:

deploy:
  queue: my-queue-1
  script: deploy

This can help us implement priority runners / priority queues and reduce complexity of the tags matching system we are using currently.

Initial discussion

The idea was introduced by @fabiopitino in #354343 (comment 1350952887):

On the argument about small vs large amount of queues. I think that the queues today are dictated by the jobs, not the runners (for the point that builds can be enqueued before a runner exists). In this case the job would go to a queue together with all other jobs having the exact same configuration. The runner, once connected would have many queues to pick jobs from. This is a problem because want to make the job picking faster instead.

The opposite is also a problem because we would have to duplicate the job for all possible runners.

I'm wondering whether we could introduce an explicit runner pool concept that users managing runners can use to associate runner instances to a queue. A runner pool comes with a queue. When a job needs to be enqueue we check all matching runners and push the job to their runner pool. If a job matches multiple shared runners but all these shared runners are assigned to the same runner pool we essentially queue the job once.

Similarly an organization could group together all their group-level runners under the same pool. Now, when a group runner pops a job from the queue it will automatically disappear for the other runners connected to the same pool.

You could pool together runners having the same set of tags, or by a subset of tags, or by project_id. The user has the flexibility of organizing the runner pools as they want.

If a runner cannot pick the first job from the queue (e.g. not matching tags) it skips it (leaving in in the queue) and fetches the next one.

Queue sizes could be limited with a ticketing system as @ayufan mentioned.

If a runner is not assigned to a pool, we use the legacy/existing queueing system. Over time we can deprecate the legacy (database) queues, for example requiring new runners to be associated to a pool, and remove the legacy queues. We could also dogfood runner pools with shared runners only or GitLab dedicated runners.

Edited by Fabio Pitino