All jobs running on same node despite others being available

Summary

Kubernetes runner is configured to run:

  • itself on node pool 1 with nodes labelled apps:true for stability of it's availability
  • concurrent pods on node pool with node labelled cicd:true for cheap preemptible runtime

Given that the concurrent pods running on the cicd node pool with 3 nodes available - I am am seeing all jobs (upto 30) all running on 1 node and the other two completely unused.

Steps to reproduce

Execute multiple jobs on a runner configured something like:

concurrent: 30
nodeSelector: {apps: "true"}
runners.nodeSelector: {cicd: "true"}

Actual behavior

image

Expected behavior

I would expect some distribution of jobs across the three available nodes. But instead I see all jobs even when upto 20+ concurrent always arriving on a single node.