Kubernetes executor concurrent config not effective after config reload via kill -s SIGHUP 1

Summary

In order to (automatically) scale a build cluster down over night and weekend and back up during the day, we modify the concurrent setting in config.toml (stored in the gitlab-runner config map) and send a SIGHUP to the runner.

Steps to reproduce

Scale down:

kubectl get cm gitlab-runner -o yaml -n gitlab-runner | sed -e 's/concurrent = 9/concurrent = 1/' | kubectl apply -f -
kubectl -n gitlab-runner exec -i $(kubectl get pods -l 'app=gitlab-runner' -n gitlab-runner -o jsonpath='{.items[0].metadata.name}') -- gitlab-runner -- kill -s SIGHUP 1

Scale up: Same as above, only concurrent is changed from 1 to 9 instead.

Actual behavior

In the morning, after setting the value from 1 to 9 and sending the signal, the runner logs 'Configuration loaded' but still doesn't run more than one job in parallel, remaining jobs are pending until each single job is done. The config.toml is definitely updated, verified by opening a shell in the pod and running less /configmaps/config.toml.

Expected behavior

After reloading the configuration, the runner should start up to 9 pending jobs in parallel again.

Relevant logs and/or screenshots

job log

...
Configuration loaded                                builds=1
...

Environment description

Kubernetes executor on AWS EKS cluster, Kubernetes version 1.23

Runtime platform                                    arch=amd64 os=linux pid=14 revision=d540b510 version=15.9.1

config.toml contents

Day:

    concurrent = 9
    check_interval = 30
    log_level = "info"
    listen_address = ':9252'

Night:

    concurrent = 1
    check_interval = 30
    log_level = "info"
    listen_address = ':9252'

Used GitLab Runner version

Version:      15.9.1
Git revision: d540b510
Git branch:   15-9-stable
GO version:   go1.18.10
Built:        2023-02-20T21:03:05+0000
OS/Arch:      linux/amd64