Sidekiq Pods take a seemingly long time to transition to state Ready
Summary
It seems that the Readiness probes take a long time to report that the sidekiq service is ready, in some cases over 1 minute. This is in addition to the already lengthy start times of the Pod. Utilize this issue to investigate the cause of Pods taking such a long time. Determine if there are any improvements that can be made. This has a direct impact on the ability to quickly scale and quickly roll out new versions of the application.
Steps to reproduce
- Kill a running sidekiq Pod
- Watch the replacement Pod take over 1 minute to become ready.
Current behavior
Long transition times between init and Ready. 71 seconds the example noted below.
Expected behavior
Start up times less than 1 minute.
Versions
- Chart:
3.1.3
- Platform:
- Cloud: GKE
- Kubernetes: (
kubectl version
)- Client:
1.14.7
- Server:
1.14.10-gke.17
- Client:
- Helm: (
helm version
)- Client:
2.14.2
- Server: n/a
- Client:
Relevant logs
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
<SNIP>
Normal Created 77s kubelet, gke-pre-gitlab-gke-node-pool-0-715a725e-kmxd Created container sidekiq
Normal Started 76s kubelet, gke-pre-gitlab-gke-node-pool-0-715a725e-kmxd Started container sidekiq
Warning Unhealthy 36s kubelet, gke-pre-gitlab-gke-node-pool-0-715a725e-kmxd Liveness probe failed: Get http://10.235.3.15:8083/liveness: dial tcp 10.235.3.15:8083: connect: connection refused
Warning Unhealthy 31s (x5 over 71s) kubelet, gke-pre-gitlab-gke-node-pool-0-715a725e-kmxd Readiness probe failed: Get http://10.235.3.15:8083/readiness: dial tcp 10.235.3.15:8083: connect: connection refused