Sidekiq gets restarted
Summary
After about 2 days of work my sidekiq pods have restarted about 50 times. That's not really good cause sometimes they restarts synchronously. Also when pod have such restarts count kubelet waits some time before restart (even if there's a lot of restarts but last was many time ago) - that makes instance have big sidekiq queues.
Steps to reproduce
Launch default configuration and heavily use instance for a while.
Configuration used
Default configuration
Current behavior
If I right understood (gitlab-org/gitlab#34547 (comment 238002701)) the problem is that when we are achieving memory killer rss limit sidekiq process gets killed gracefully. But suddenly container doesn't have any supervisor and startup script (/scripts/process-wrapper
) doesn't have any restart logic. So when we achieving Memory killer limit container gets restarted.
This thing have multiple cons:
- For most k8s admins container restarts is evidence of some problem with application
- Most of default alerting rules for clusters have alerts for pods with big restarts count
- When restarts count gets bigger kubelet waits some time before restart that could make sidekiq queues have big latency (if sidekiq workers restart crossed in time)
Expected behavior
Sidekiq process is restarted without container restart
Versions
- Chart: 2.4.0
- Platform:
- Self-hosted: Kubernetes
- Kubernetes: (
kubectl version
)- Client: 1.16.2
- Server: 1.14.3
- Helm: (
helm version
)- Client: 12.3.1
- Server: 12.3.1