Kubernetes runner livenessProbe not recognizing dead runner
We are using the v0.2.0 Helm chart to deploy Gitlab Runners. Every few days, a log message ends up in the Gitlab runner with something like is not healthy and will be disabled!
. However, the livenessProbe
only checks if the process is running, not if it's actually healthy. The only solution at this point is to manually delete the Pod
and then the runner is working again.
This error message is outputted here: https://gitlab.com/gitlab-org/gitlab-runner/blob/master/commands/health_helper.go#L65.
We have a custom patched version of the Helm chart that tee
s the runner logs, and we've updated the livenessProbe
to grep
for that message. While this works for us today, it's not the greatest solution.
I'd be happy to contribute a better solution if I was pointed in the right direction.