Use RssMemoryLimit monitor for Sidekiq
What does this MR do and why?
In #379198 (closed) we have introduced RssMemoryLimit as a backward-compatible Watchdog monitor for Puma.
Since Sidekiq Daemon Memory Killer is also monitoring RSS with a fixed memory budget, we could re-use the RssMemoryLimit monitor and configure it to be used for Sidekiq as well. This will allow us to move away from Sidekiq Memory killer and use only Watchdog for both Puma and Sidekiq. (More info here and here)
Watchdog with RssMemoryLimit monitor vs Sidekiq Memory Killer:
- RssMemoryLimit doesn't have soft and hard limits.
- RssMemoryLimit uses the maximum number of strikes instead of Grace time.
This MR configures Watchdog for Sidekiq to use two RssMemoryLimit monitors. It will use same env variables the current Sidekiq daemon memory killer is using for backward compatibility.
- It configures one RssMemoryLimit monitor with:
-
memory_limit=SIDEKIQ_MEMORY_KILLER_MAX_RSS(soft limit) -
max_strikes=SIDEKIQ_MEMORY_KILLER_GRACE_TIME/SIDEKIQ_MEMORY_KILLER_CHECK_INTERVAL(if the soft limit is exceeded, wait forgrace_time/sleep_time_intervalnumber of times before restarting the process)
-
- It configures the second RssMemoryLimit monitor with:
-
memory_limit=SIDEKIQ_MEMORY_KILLER_HARD_LIMIT_RSS(hard limit) -
max_strikes= 0 (If the hard limit is exceeded, we restart the process immediately)
-
At the moment, this does not affect production or self-managed customers since the Watchdog is disabled by default for Sidekiq.
This is just a prerequisite to replace the Sidekiq Daemon memory killer with the Watchdog.
Screenshots or screen recordings
Screenshots are required for UI changes and strongly recommended for all other merge requests.
How to set up and validate locally
- Enable the environment variables
export GITLAB_MEMORY_WATCHDOG_ENABLED=true export SIDEKIQ_MEMORY_KILLER_MAX_RSS=1000000 export SIDEKIQ_MEMORY_KILLER_HARD_LIMIT_RSS=1500000 export SIDEKIQ_MEMORY_KILLER_GRACE_TIME=300 export SIDEKIQ_MEMORY_KILLER_CHECK_INTERVAL=3 - In the rails console enable the watchdog monitoring
Feature.enable(:gitlab_memory_watchdog) - In the rails console enable the handler that will restart the sidekiq process
Feature.enable(:enforce_memory_watchdog) - Click the logs and metrics
gdk tail rails-background-jobs
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.
Related to: #382538 (closed)