Have a runner not picking up a job if it doesn't have enough disk space/CPU/Memory available
Description
We would like to be able to specify in the .gitlab-ci.yml file itself the required resources for a runner to pick up a job.
We have a pipeline(let's called it pipeline-1
) that runs up to 6 times per minute, takes between 7/8 minutes and temporarily generating huge docker images. As we don't have much control over which runner picks up jobs, it happens that the same runner runs multiple pipeline-1
at the same time(even though we have a decent amount of available runners), which causes the runner to run out of disk space and the pipelines to fail.
The first workaround would be to have dedicated runners for this pipeline with a concurrent set to 1 but it would be a huge waste of resource as we would need a minimum of 42 runners just for that pipeline (triggered 6 times per minute, takes 7 minutes to run).
The second workaround would be to increase the disk space on our runners but it's not a real solution for us as this pipeline could be triggered more and more often.
The best solution would be to have the runner check for its available resources before claiming the job.
Proposal
We could have the option in the .gitlab-ci.yml
file to specify the minimum resources a runner should have available to pick up a job.
Maybe something like:
resources:
- disk: 20G
- cpu: 30%
...etc
Links to related issues and merge requests / references
zendesk ticket: https://support.gitlab.com/hc/en-us/requests/113006