Add a helm chart configuration option to specify resource requests and limits of created job pods
Initially I thought that the resources configuration in the helm chart was for the created job pods. Upon investigation, this is not the case and it is only there for the scheduler pod that creates the job pods.
Not being able to define resource requests and limits for the created job pods results in highly concentrated and unresponsive nodes and prevents autoscaling from working when running many heavy jobs in a pipeline.
The only solution right now would be to increase the minimum node count of the cluster which leads to massively increased costs at scale. This would still not prevent nodes from becoming highly over concentrated because the pod scheduler is not taking into account the resources that the pods need.
Please, please, please add a helm configuration option to allow normal cluster operation.
runners:
resources:
{}
# requests:
# cpu: 500m
# memory: 1Gi
# limits:
# cpu: 750m
# memory: 1.5Gi