Skip to content

Log running jobs for Sidekiq memory killer

What does this MR do and why?

It adds currently running jobs in the log. Once when RSS is above Soft limit in Sidekiq Memory Killer, we now log that we are over the soft limit, and once deadline expires, we start terminating the process.

How to set up and validate locally

  1. Make sure that environment variable SIDEKIQ_DAEMON_MEMORY_KILLER is set to 1
  2. Make sure to configure the export SIDEKIQ_MEMORY_KILLER_MAX_RSS=300000 to some really small value that would be easily reached by the local environment
  3. Run long running job (You can add sleep 3000)
  4. gdk tail rails-background-jobs and check if the log contains running_jobs
{
   "severity":"WARN",
   "time":"2022-07-15T11:11:35.716Z",
   "class":"Gitlab::SidekiqDaemon::MemoryKiller",
   "pid":49580,
   "message":"Sidekiq worker RSS out of range",
   "current_rss":681884,
   "soft_limit_rss":300000,
   "hard_limit_rss":274877906944,
   "reason":"current_rss(681884) \u003e soft_limit_rss(100000)",
   "running_jobs":[
      {
         "jid":"ffbf12ed42c36360e492a620",
         "worker_class":"UpdateAllMirrorsWorker"
      },
      {
         "jid":"5eb482e553a6c34aab1455cc",
         "worker_class":"UpdateAllMirrorsWorker"
      },
      {
         "jid":"cefe48fab1537c812bfa78ef",
         "worker_class":"UpdateAllMirrorsWorker"
      }
   ],
   "retry":0

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Related to: #367668 (closed)

Edited by Nikola Milojevic

Merge request reports