Sidekiq Memory Killer not working
Summary
We are facing an issue where the Sidekiq Memory Killer kicks in, but doesnt kill the process.
In the logs we can see that its logging 'waiting 6 seconds...', but its never logging the warning without the 'waiting' part.
Looks like the thread is killing itself, and thus not reaching the SIGKILL process. The problem is, the process never quits and needs to be manually killed before processing any new jobs
Steps to reproduce
Example Project
What is the current bug behavior?
Sidekiq does not kill itself
What is the expected correct behavior?
Sidekiq should quit & be restarted when the Memory Killer kills it
Relevant logs and/or screenshots
{"severity":"WARN","time":"2019-10-17T13:56:04.825Z","class":"ExpirePipelineCacheWorker","pid":18924,"signal":"SIGKILL","message":"waiting 6 seconds before sending Sidekiq worker PGRP-18924 SIGKILL (die)"}
{"severity":"INFO","time":"2019-10-17T13:56:04.825Z","message":"Shutting down"}
{"severity":"INFO","time":"2019-10-17T13:56:05.326Z","message":"Pausing to allow workers to finish..."}
{"severity":"WARN","time":"2019-10-17T13:56:08.329Z","message":"Terminating 16 busy worker threads"}
{"severity":"INFO","time":"2019-10-17T13:56:08.340Z","message":"Pushed 16 jobs back to Redis"}
{"severity":"INFO","time":"2019-10-17T13:56:08.348Z","message":"Bye!"}
Manually killing the pid
2019-10-17T14:10:10.141Z 38758 TID-ov9sh5yh6 INFO: Booting Sidekiq 5.2.7 with redis options {:scheme=>"redis", :host=>"gitlab-redis", :port=>6379, :password=>"REDACTED", :sentinels=>[{:host=>"REDACTED", :port=>26379}, {:host=>"REDACTED", :port=>26379}, {:host=>"REDACTED", :port=>26379}], :namespace=>"resque:gitlab", :id=>"Sidekiq-server-PID-38758", :url=>nil}
Output of checks
Results of GitLab environment info
Results of GitLab application Check
Possible fixes
Looks like sleep is getting interrupted