Runner becomes unavailable intermittently - appears to deregister and fails to create jobs

In the past week our GitLab shared runner deployed on Kubernetes has become unavailable twice. The pod of the runner remains in a ready state but it becomes unable to create jobs as it appears to no longer be registered. When trying to run a pipeline, this message appears in gitlab: This job is stuck, because you don't have any active runners that can run this job. Go to Runners page.

This has also happened one other time in the past. It doesn't appear to be an issue with our registration token because we are otherwise able to use the runner successfully.

To fix the issue, we delete the pod of the shared runner and a new one will come up and register with gitlab with no problem. I have not been able to reproduce this as it is happens intermittently with no clear indication of what could be triggering it. These are the logs from the pod before it is deleted:

WARNING: Running in user-mode.
WARNING: The user-mode requires you to manually start builds processing:
WARNING: $ gitlab-runner run
WARNING: Use sudo for system-mode:
WARNING: $ sudo gitlab-runner...

Registering runner... succeeded                     runner=KKEc2fHk
Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!
total 12
drwxrwsrwx. 2 root                  59417 4096 Dec 10 06:11 .
drwxr-xr-x. 3 gitlab-runner gitlab-runner 4096 Dec 13 01:37 ..
-rw-------. 1         59417         59417 1251 Dec 13 01:37 config.toml
Starting multi-runner from /home/gitlab-runner/.gitlab-runner/config.toml ...  builds=0
WARNING: Running in user-mode.
WARNING: Use sudo for system-mode:
WARNING: $ sudo gitlab-runner...

Configuration loaded                                builds=0
WARNING: 'metrics_server' configuration entry is deprecated and will be removed in one of future releases; please use 'listen_address' instead
Metrics server listening                            address=0.0.0.0:9252 builds=0
Listen address not defined, session server disabled  builds=0
ERROR: Checking for jobs... forbidden               runner=c24fc342
ERROR: Checking for jobs... forbidden               runner=c24fc342
ERROR: Checking for jobs... forbidden               runner=c24fc342
ERROR: Runner https://gitlab.mycompany.comac123bce is not healthy and will be disabled!
ERROR: Runner https://gitlab.mycompany.comac123bce is not healthy, but will be checked!
ERROR: Checking for jobs... forbidden               runner=c24fc342
ERROR: Checking for jobs... forbidden               runner=c24fc342
ERROR: Checking for jobs... forbidden               runner=c24fc342
ERROR: Runner https://gitlab.mycompany.comac123bce is not healthy and will be disabled!
ERROR: Runner https://gitlab.mycompany.comac123bce is not healthy, but will be checked!
ERROR: Checking for jobs... forbidden               runner=c24fc342
ERROR: Checking for jobs... forbidden               runner=c24fc342
ERROR: Checking for jobs... forbidden               runner=c24fc342
ERROR: Runner https://gitlab.mycompany.comac123bce is not healthy and will be disabled!
ERROR: Runner https://gitlab.mycompany.comac123bce is not healthy, but will be checked!
ERROR: Checking for jobs... forbidden               runner=c24fc342
ERROR: Checking for jobs... forbidden               runner=c24fc342
ERROR: Checking for jobs... forbidden               runner=c24fc342
ERROR: Runner https://gitlab.mycompany.comac123bce is not healthy and will be disabled!
ERROR: Runner https://gitlab.mycompany.comac123bce is not healthy, but will be checked!
ERROR: Checking for jobs... forbidden               runner=c24fc342
ERROR: Checking for jobs... forbidden               runner=c24fc342
ERROR: Checking for jobs... forbidden               runner=c24fc342
ERROR: Runner https://gitlab.mycompany.comac123bce is not healthy and will be disabled!
ERROR: Runner https://gitlab.mycompany.comac123bce is not healthy, but will be checked!
ERROR: Checking for jobs... forbidden               runner=c24fc342
ERROR: Checking for jobs... forbidden               runner=c24fc342
ERROR: Checking for jobs... forbidden               runner=c24fc342
ERROR: Runner https://gitlab.mycompany.comac123bce is not healthy and will be disabled!
ERROR: Runner https://gitlab.mycompany.comac123bce is not healthy, but will be checked!
ERROR: Checking for jobs... forbidden               runner=c24fc342
ERROR: Checking for jobs... forbidden               runner=c24fc342
ERROR: Checking for jobs... forbidden               runner=c24fc342
ERROR: Runner https://gitlab.mycompany.comac123bce is not healthy and will be disabled!
ERROR: Runner https://gitlab.mycompany.comac123bce is not healthy, but will be checked!
ERROR: Checking for jobs... forbidden               runner=c24fc342
ERROR: Checking for jobs... forbidden               runner=c24fc342
ERROR: Checking for jobs... forbidden               runner=c24fc342
ERROR: Runner https://gitlab.mycompany.comac123bce is not healthy and will be disabled!
ERROR: Runner https://gitlab.mycompany.comac123bce is not healthy, but will be checked!
ERROR: Checking for jobs... forbidden               runner=c24fc342
ERROR: Checking for jobs... forbidden               runner=c24fc342
ERROR: Checking for jobs... forbidden               runner=c24fc342
ERROR: Runner https://gitlab.mycompany.comac123bce is not healthy and will be disabled!
ERROR: Runner https://gitlab.mycompany.comac123bce is not healthy, but will be checked!
ERROR: Checking for jobs... forbidden               runner=c24fc342
ERROR: Checking for jobs... forbidden               runner=c24fc342
ERROR: Checking for jobs... forbidden               runner=c24fc342
ERROR: Runner https://gitlab.mycompany.comac123bce is not healthy and will be disabled!
ERROR: Runner https://gitlab.mycompany.comac123bce is not healthy, but will be checked!
ERROR: Checking for jobs... forbidden               runner=c24fc342
ERROR: Checking for jobs... forbidden               runner=c24fc342
ERROR: Checking for jobs... forbidden               runner=c24fc342
ERROR: Runner https://gitlab.mycompany.comac123bce is not healthy and will be disabled!
ERROR: Runner https://gitlab.mycompany.comac123bce is not healthy, but will be checked!
ERROR: Checking for jobs... forbidden               runner=c24fc342
ERROR: Checking for jobs... forbidden               runner=c24fc342
ERROR: Checking for jobs... forbidden               runner=c24fc342
ERROR: Runner https://gitlab.mycompany.comac123bce is not healthy and will be disabled!
ERROR: Runner https://gitlab.mycompany.comac123bce is not healthy, but will be checked!
ERROR: Checking for jobs... forbidden               runner=c24fc342
ERROR: Checking for jobs... forbidden               runner=c24fc342
ERROR: Checking for jobs... forbidden               runner=c24fc342
ERROR: Runner https://gitlab.mycompany.comac123bce is not healthy and will be disabled!
ERROR: Runner https://gitlab.mycompany.comac123bce is not healthy, but will be checked!
ERROR: Checking for jobs... forbidden               runner=c24fc342
ERROR: Checking for jobs... forbidden               runner=c24fc342
ERROR: Checking for jobs... forbidden               runner=c24fc342
ERROR: Runner https://gitlab.mycompany.comac123bce is not healthy and will be disabled!
ERROR: Runner https://gitlab.mycompany.comac123bce is not healthy, but will be checked!
ERROR: Checking for jobs... forbidden               runner=c24fc342
ERROR: Checking for jobs... forbidden               runner=c24fc342
ERROR: Checking for jobs... forbidden               runner=c24fc342
ERROR: Runner https://gitlab.mycompany.comac123bce is not healthy and will be disabled!
ERROR: Runner https://gitlab.mycompany.comac123bce is not healthy, but will be checked!
ERROR: Checking for jobs... forbidden               runner=c24fc342
ERROR: Checking for jobs... forbidden               runner=c24fc342
ERROR: Checking for jobs... forbidden               runner=c24fc342
ERROR: Runner https://gitlab.mycompany.comac123bce is not healthy and will be disabled!
ERROR: Runner https://gitlab.mycompany.comac123bce is not healthy, but will be checked!
ERROR: Checking for jobs... forbidden               runner=c24fc342
ERROR: Checking for jobs... forbidden               runner=c24fc342
ERROR: Checking for jobs... forbidden               runner=c24fc342
ERROR: Runner https://gitlab.mycompany.comac123bce is not healthy and will be disabled!
ERROR: Runner https://gitlab.mycompany.comac123bce is not healthy, but will be checked!
ERROR: Checking for jobs... forbidden               runner=c24fc342
ERROR: Checking for jobs... forbidden               runner=c24fc342
ERROR: Checking for jobs... forbidden               runner=c24fc342
ERROR: Runner https://gitlab.mycompany.comac123bce is not healthy and will be disabled!
ERROR: Runner https://gitlab.mycompany.comac123bce is not healthy, but will be checked!
ERROR: Checking for jobs... forbidden               runner=c24fc342
ERROR: Checking for jobs... forbidden               runner=c24fc342
ERROR: Checking for jobs... forbidden               runner=c24fc342
ERROR: Runner https://gitlab.mycompany.comac123bce is not healthy and will be disabled!
ERROR: Runner https://gitlab.mycompany.comac123bce is not healthy, but will be checked!
ERROR: Checking for jobs... forbidden               runner=c24fc342
ERROR: Checking for jobs... forbidden               runner=c24fc342
ERROR: Checking for jobs... forbidden               runner=c24fc342
ERROR: Runner https://gitlab.mycompany.comac123bce is not healthy and will be disabled!
ERROR: Runner https://gitlab.mycompany.comac123bce is not healthy, but will be checked!
ERROR: Checking for jobs... forbidden               runner=c24fc342
ERROR: Checking for jobs... forbidden               runner=c24fc342
ERROR: Checking for jobs... forbidden               runner=c24fc342
ERROR: Runner https://gitlab.mycompany.comac123bce is not healthy and will be disabled!
ERROR: Runner https://gitlab.mycompany.comac123bce is not healthy, but will be checked!
ERROR: Checking for jobs... forbidden               runner=c24fc342
ERROR: Checking for jobs... forbidden               runner=c24fc342
ERROR: Checking for jobs... forbidden               runner=c24fc342
ERROR: Runner https://gitlab.mycompany.comac123bce is not healthy and will be disabled!