I'm using have my GitLab CI runner running in a Kubernetes cluster and I'm using the Kubernetes executor for the jobs. Let's say I have 2 nodes and I want to be sure, that it's possible to run 2 jobs in parallel, but not on the same node. If I add a third node, also the limit of parallel jobs should be increased to 3 (if possible, in an automatical way). I'm aware of the general concurrent parameter, but that's only the global limit of parallel jobs, not the "jobs per node" (I'm searching for). Is it possible, maybe with labels/node selector annotations, to get this behaviour? Thanks in advance for your help!
CI build jobs can be highly parallelized and utilize all resources of the node it runs on. What I'm seeing now is that one node is processing two jobs simultaneously (of the same pipeline) which are fighting for resources, while there are other nodes available, doing almost nothing.
Let's say I have 2 nodes and I want to be sure, that it's possible to run 2 jobs in parallel, but not on the same node.
What is the use case here? Mind if you give a concrete example why you would need to do this? It's up to kubernets to schdule the pod in the node that it finds suitable, not sure how we can control this.
Is it possible, maybe with labels/node selector annotations, to get this behaviour?
We do have support for node_selectors in our config file, but this would be on a runner level not on a job level, it's an interesting ~feature if we can have in out yaml to have node selectors there, but will mean that we have to extend our yaml definition, thoughts @brendan ?
The use case is simple: CI build jobs can be highly parallelized and utilize all resources of the node it runs on. What I'm seeing now is that one node is processing two jobs simultaneously (of the same pipeline) which are fighting for resources, while there are other nodes available, doing almost nothing.
What I'm seeing now is that one node is processing two jobs simultaneously (of the same pipeline) which are fighting for resources, while there are other nodes available, doing almost nothing
@jansmets That is up to the kubernetes bin-packing algorithm, I don't think we should do anything about that, we should let kubernetes do what it's best at.
In a way, I agree, but this still doesn't solve the problem that eventually there can be multiple jobs running per node.
What we would like is gitlab-runner to behave like a daemonset where each set is configured with concurrency=1 and a nodeSelector tying it to each specific node.
Perhaps there are other ways to do this (taint/tolerations?) but I don't think gitlab-runner supports these yet.