Sidekiq Pods take a seemingly long time to transition to state Ready

Summary

It seems that the Readiness probes take a long time to report that the sidekiq service is ready, in some cases over 1 minute. This is in addition to the already lengthy start times of the Pod. Utilize this issue to investigate the cause of Pods taking such a long time. Determine if there are any improvements that can be made. This has a direct impact on the ability to quickly scale and quickly roll out new versions of the application.

Steps to reproduce

  1. Kill a running sidekiq Pod
  2. Watch the replacement Pod take over 1 minute to become ready.

Current behavior

Long transition times between init and Ready. 71 seconds the example noted below.

Expected behavior

Start up times less than 1 minute.

Versions

  • Chart: 3.1.3
  • Platform:
    • Cloud: GKE
  • Kubernetes: (kubectl version)
    • Client: 1.14.7
    • Server: 1.14.10-gke.17
  • Helm: (helm version)
    • Client: 2.14.2
    • Server: n/a

Relevant logs

Events:
  Type     Reason     Age                From                                                   Message
  ----     ------     ----               ----                                                   -------
<SNIP>
  Normal   Created    77s                kubelet, gke-pre-gitlab-gke-node-pool-0-715a725e-kmxd  Created container sidekiq
  Normal   Started    76s                kubelet, gke-pre-gitlab-gke-node-pool-0-715a725e-kmxd  Started container sidekiq
  Warning  Unhealthy  36s                kubelet, gke-pre-gitlab-gke-node-pool-0-715a725e-kmxd  Liveness probe failed: Get http://10.235.3.15:8083/liveness: dial tcp 10.235.3.15:8083: connect: connection refused
  Warning  Unhealthy  31s (x5 over 71s)  kubelet, gke-pre-gitlab-gke-node-pool-0-715a725e-kmxd  Readiness probe failed: Get http://10.235.3.15:8083/readiness: dial tcp 10.235.3.15:8083: connect: connection refused