Configurable liveness and readiness probes
We are currently seeing restarts of the Viking controller due to liveness probe failures. Our current working hypothesis is that this might be caused by us updating to Kubernetes 1.20 recently which included the following note in its changelog:
A bug was fixed in kubelet where exec probe timeouts were not respected. This may result in unexpected behavior since the default timeout (if not specified) is 1s which may be too small for some exec probes. Ensure that pods relying on this behavior are updated to correctly handle probe timeouts. See configure probe section of the documentation for more details.
A common pattern in other charts is to have the liveness and readiness probe parameters configurable because the performance of the probes can differ depending on the cluster they are run on, e. g.:
https://github.com/bitnami/charts/blob/master/bitnami/postgresql/templates/statefulset.yaml#L389
It would be nice if we could add this to the Chart here, to make these settings easier to tweak. It might be necessary to also adjust the default timeoutSeconds
if it indeed turns out to be a general problem on Kubernetes 1.20.