Use RssMemoryLimit monitor for Sidekiq

What does this MR do and why?

In #379198 (closed) we have introduced RssMemoryLimit as a backward-compatible Watchdog monitor for Puma.

Since Sidekiq Daemon Memory Killer is also monitoring RSS with a fixed memory budget, we could re-use the RssMemoryLimit monitor and configure it to be used for Sidekiq as well. This will allow us to move away from Sidekiq Memory killer and use only Watchdog for both Puma and Sidekiq. (More info here and here)

Watchdog with RssMemoryLimit monitor vs Sidekiq Memory Killer:

  • RssMemoryLimit doesn't have soft and hard limits.
  • RssMemoryLimit uses the maximum number of strikes instead of Grace time.

This MR configures Watchdog for Sidekiq to use two RssMemoryLimit monitors. It will use same env variables the current Sidekiq daemon memory killer is using for backward compatibility.

  1. It configures one RssMemoryLimit monitor with:
    • memory_limit = SIDEKIQ_MEMORY_KILLER_MAX_RSS (soft limit)
    • max_strikes = SIDEKIQ_MEMORY_KILLER_GRACE_TIME/SIDEKIQ_MEMORY_KILLER_CHECK_INTERVAL (if the soft limit is exceeded, wait for grace_time/sleep_time_interval number of times before restarting the process)
  2. It configures the second RssMemoryLimit monitor with:
    • memory_limit = SIDEKIQ_MEMORY_KILLER_HARD_LIMIT_RSS (hard limit)
    • max_strikes = 0 (If the hard limit is exceeded, we restart the process immediately)

At the moment, this does not affect production or self-managed customers since the Watchdog is disabled by default for Sidekiq.

This is just a prerequisite to replace the Sidekiq Daemon memory killer with the Watchdog.

Screenshots or screen recordings

Screenshots are required for UI changes and strongly recommended for all other merge requests.

How to set up and validate locally

  1. Enable the environment variables
      export GITLAB_MEMORY_WATCHDOG_ENABLED=true
      export SIDEKIQ_MEMORY_KILLER_MAX_RSS=1000000
      export SIDEKIQ_MEMORY_KILLER_HARD_LIMIT_RSS=1500000
      export SIDEKIQ_MEMORY_KILLER_GRACE_TIME=300
      export SIDEKIQ_MEMORY_KILLER_CHECK_INTERVAL=3
  2. In the rails console enable the watchdog monitoring
    Feature.enable(:gitlab_memory_watchdog)
  3. In the rails console enable the handler that will restart the sidekiq process
    Feature.enable(:enforce_memory_watchdog)
  4. Click the logs and metrics gdk tail rails-background-jobs

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Related to: #382538 (closed)

Edited by Aleksei Lipniagov

Merge request reports

Loading
Loading