scale kubernetes executor concurrency to available nodes
I'm running a bare metal cluster with a fixed amount of nodes.
I've modified gitlab-runner to include a node affinity (requiredDuringSchedulingIgnoredDuringExecution) in the pod spec to guarantee that I don't run more than one job per node.
Is there a way I can dynamically set/modify the concurrency of a runner, ideally I want the concurrency to match the number of nodes in the cluster (or better the number of nodes that have some specific label).
Setting the concurrency# higher than #node will put set gitlab jobs in a Running stage, but Pending on k8s.
If there a way this can be accomplished with HPA ? Ideally there would be a runners API that would allow changing the concurrency level. (I'll make this feature request in the gitab-runner repo)
And I believe helm terminates the runner pod when it has an updated configuration. Ideally only the ConfigMap should be modified (which causes gitlab-runner to auto reload) Thank you