Use RssMemoryLimit Watchdog monitor for Sidekiq

In #379198 (closed) we have introduced RssMemoryLimit as a backward-compatible Watchdog monitor for Puma.

Since Sidekiq Daemon Memory Killer is also monitoring RSS with a fixed memory budget, we could re-use the RssMemoryLimit monitor and configure it to be used for Sidekiq as well. This will allow us to move away from Sidekiq Memory killer and use only Watchdog for both Puma and Sidekiq. (More info [here](&8415 (closed) and here)

Sidekiq Daemon MemoryKiller is using the following environment variables for configuration:

  • SIDEKIQ_MEMORY_KILLER_MAX_RSS - RSS below this soft limit is considered safe
  • SIDEKIQ_MEMORY_KILLER_HARD_LIMIT_RSS - if RSS exceeds this hard limit, the process will be gracefully terminated
  • SIDEKIQ_MEMORY_KILLER_GRACE_TIME - RSS exceeding the above soft limit is allowed for the GRACE period of time, when the grace period is over, the process is restarted gracefully.
  • SIDEKIQ_MEMORY_KILLER_CHECK_INTERVAL - Time interval to check for RSS (sleep time interval)

Differences

  • RssMemoryLimit doesn't have a soft and hard limits.
  • RssMemoryLimit uses maximum number of strikes instead of Grace time.

Proposed solution

  1. Configure one RssMemoryLimit monitor with:
    • memory_limit = SIDEKIQ_MEMORY_KILLER_MAX_RSS (soft limit)
    • max_strikes = SIDEKIQ_MEMORY_KILLER_GRACE_TIME/SIDEKIQ_MEMORY_KILLER_CHECK_INTERVAL (if the soft limit is exceeded, wait for grace_time/sleep_time_interval number of times before restarting the process)
  2. Configure second RssMemoryLimit monitor with:
    • memory_limit = SIDEKIQ_MEMORY_KILLER_HARD_LIMIT_RSS (hard limit)
    • max_strikes = 0 (If hard limit is exceeded, we restart process immediately)

At the moment, this would not affect production or self-managed customers since the Watchdog is disabled by default for Sidekiq.

This is just a prerequisite to replace the Sidekiq Daemon memory killer with the Watchdog.

Edited by Nikola Milojevic