Sidekiq memory killer potentially kills innocent jobs
Zendesk: https://gitlab.zendesk.com/agent/tickets/16872
Sidekiq memory killer may cause innocent jobs to killed and won't put them back in the queue (or register 'fail' which would cause Sidekiq to retry).
Sidekiq has a GRACE_TIME
. If memory exceeds the threshold the killer will wait GRACE_TIME
to issue the shutdown. This is great for long running jobs that need to finish up. However, as far as we understand, if a long-running jobs finishes within GRACE_TIME
, another job could be picked up. Then, after GRACE_TIME
passes the job is unceremoniously terminated.
Customer wonders if we should be terminating Sidekiq in the same manner as the Sidekiq gem does it. It re-enqueues the jobs, or marks them as 'failed' so they are retried.
cc/ @jacobvosmaer Pinging you because this is with a customer we've been discussing the memory killer with.