Name the Sidekiq settings behind memwd_max_rss_bytes

What does this MR do?

The "Monitor worker restarts" section of the Sidekiq memory killer page says that memwd_max_rss_bytes is "the RSS limit set through per_worker_max_memory_mb". That is the Puma setting: in the Linux package, puma['per_worker_max_memory_mb'] becomes PUMA_WORKER_MAX_MEMORY, which only the Puma watchdog reads. It does nothing for Sidekiq, so an administrator who follows the sentence changes a limit that the Sidekiq log line does not come from.

For Sidekiq, Gitlab::Memory::Watchdog::Configurator.configure_for_sidekiq builds one RSS monitor for each limit that is set: the soft limit from SIDEKIQ_MEMORY_KILLER_MAX_RSS and the hard limit from SIDEKIQ_MEMORY_KILLER_HARD_LIMIT_RSS, both converted from kilobytes to bytes. The monitor that fires logs its own limit as memwd_max_rss_bytes. The first commit replaces the sentence with that. The two variables are already described further up the same page.

The second commit fixes memwd_handler_class in the same example log line, and in the matching example on the Puma page. They show Gitlab::Memory::Watchdog::SidekiqHandler and Gitlab::Memory::Watchdog::PumaHandler, and GitLab defines neither class. The watchdog logs handler.class.name, and the handlers the configurator installs are Gitlab::Memory::Watchdog::Handlers::SidekiqHandler and Gitlab::Memory::Watchdog::Handlers::PumaHandler, so searching the logs for the documented value finds nothing. Handlers has held both since f70bfbac (January 2023), which moved PumaHandler into it and replaced the TermProcessHandler that Sidekiq used until then with Handlers::SidekiqHandler.

The issue also suggests a bullet for Puma. I left it out: the Sidekiq page and its example are about Sidekiq only, and the Puma page already explains the same field for Puma, correctly (per_worker_max_memory_mb or DEFAULT_PUMA_WORKER_RSS_LIMIT_MB).

I left the rest of the example log line as it is. Its "memwd_max_strikes": 5 is not what the defaults give (300 for the soft limit, 0 for the hard limit), but SIDEKIQ_MEMORY_KILLER_GRACE_TIME=15 with the default 3-second check interval produces it, together with the "memwd_sleep_time_s": 3 beside it. I can change it to the default values if you prefer. I did not touch the doc-locale copies of either page, which carry the same lines, because the documentation guidelines say translations do not take contributions yet.

Where the values come from (master b0b523cffa1)

lib/gitlab/memory/watchdog/configurator.rb, the Sidekiq monitors:

if ENV['SIDEKIQ_MEMORY_KILLER_MAX_RSS'].to_i.nonzero?
  soft_limit_bytes = ENV['SIDEKIQ_MEMORY_KILLER_MAX_RSS'].to_i.kilobytes
  grace_time = ENV.fetch('SIDEKIQ_MEMORY_KILLER_GRACE_TIME', DEFAULT_SIDEKIQ_GRACE_TIME_S).to_i
  max_strikes = grace_time / sidekiq_sleep_time

  stack.push Gitlab::Memory::Watchdog::Monitor::RssMemoryLimit,
    memory_limit_bytes: soft_limit_bytes,
    max_strikes: max_strikes.to_i,
    monitor_name: :rss_memory_soft_limit
end

if ENV['SIDEKIQ_MEMORY_KILLER_HARD_LIMIT_RSS'].to_i.nonzero?
  hard_limit_bytes = ENV['SIDEKIQ_MEMORY_KILLER_HARD_LIMIT_RSS'].to_i.kilobytes

  stack.push Gitlab::Memory::Watchdog::Monitor::RssMemoryLimit,
    memory_limit_bytes: hard_limit_bytes,
    max_strikes: 0,
    monitor_name: :rss_memory_hard_limit
end

PUMA_WORKER_MAX_MEMORY is read in one place, configure_monitors_for_puma in the same file, and config/initializers/memory_watchdog.rb applies configure_for_puma only when Gitlab::Runtime.puma? and configure_for_sidekiq only when Gitlab::Runtime.sidekiq?.

lib/gitlab/memory/watchdog/monitor/rss_memory_limit.rb puts the limit it was built with into the logged payload:

def payload(worker_rss_bytes, memory_limit_bytes)
  {
    message: 'rss memory limit exceeded',
    memwd_rss_bytes: worker_rss_bytes,
    memwd_max_rss_bytes: memory_limit_bytes
  }
end

The handler class comes from lib/gitlab/memory/watchdog.rb:

def log_labels(extra = {})
  extra.merge(
    memwd_handler_class: handler.class.name,
    memwd_sleep_time_s: sleep_time_seconds
  )
end

and the configurator sets config.handler = Gitlab::Memory::Watchdog::Handlers::PumaHandler.new and config.handler = Gitlab::Memory::Watchdog::Handlers::SidekiqHandler.new(...). Apart from these two examples and their doc-locale copies, no file in the repository contains Watchdog::SidekiqHandler or Watchdog::PumaHandler.

In omnibus-gitlab (master 12a6ea3c), files/gitlab-cookbooks/gitlab/recipes/gitlab-rails.rb has rails_env['PUMA_WORKER_MAX_MEMORY'] = node['gitlab']['puma']['per_worker_max_memory_mb'], and files/gitlab-cookbooks/gitlab/attributes/default.rb sets SIDEKIQ_MEMORY_KILLER_MAX_RSS to 2000000, the default the page gives.

The existing spec/lib/gitlab/memory/watchdog/configurator_spec.rb already expects memwd_max_rss_bytes to be SIDEKIQ_MEMORY_KILLER_MAX_RSS.kilobytes and SIDEKIQ_MEMORY_KILLER_HARD_LIMIT_RSS.kilobytes, so this MR adds no spec. To check it against the page's example, I ran a throwaway spec (not part of this MR) with fast_spec_helper that builds the Sidekiq monitors as configure_for_sidekiq does and reads the payloads, with PUMA_WORKER_MAX_MEMORY=1200 set to show it is ignored. It ran on master b00391b1ec3, whose lib/gitlab/memory, spec/lib/gitlab/memory and memory watchdog initializer are the same as on b0b523cffa1:

• SIDEKIQ_MEMORY_KILLER_MAX_RSS=614400 alone: only rss_memory_soft_limit fires, with memwd_max_rss_bytes: 629145600 (the value in the page's example) and memwd_max_strikes: 300 (the default grace time of 900 seconds over the default check interval of 3 seconds).
• SIDEKIQ_MEMORY_KILLER_HARD_LIMIT_RSS=3000000 alone: only rss_memory_hard_limit fires, with memwd_max_rss_bytes: 3072000000 and memwd_max_strikes: 0.
• Neither set: no monitor at all.
• SIDEKIQ_MEMORY_KILLER_MAX_RSS=614400 with SIDEKIQ_MEMORY_KILLER_GRACE_TIME=15, over the limit on six checks in a row: the sixth fires with memwd_max_rss_bytes: 629145600, memwd_max_strikes: 5 and memwd_cur_strikes: 6, the values of the page's example.

Gitlab::Memory::Watchdog::Configurator
  # order random
  when the soft limit is 614400 KB with a 15-second grace time
    logs the payload of the page example on the sixth violation in a row
  when neither Sidekiq limit is set
    monitors nothing, whatever the Puma limit says
  when only SIDEKIQ_MEMORY_KILLER_MAX_RSS is set to the example value 614400
    logs the soft limit in bytes, the documented example value, and ignores the Puma limit
  when only SIDEKIQ_MEMORY_KILLER_HARD_LIMIT_RSS is set
    logs the hard limit in bytes with no strikes allowed

10 examples, 0 failures

The other six examples are the existing rss_memory_limit_spec.rb and monitor_state_spec.rb, run in the same process. Expecting the Puma limit instead (1200 MB, which is 1258291200 bytes), as the current sentence implies, fails:

expected {:memwd_cur_strikes => 1, :memwd_max_rss_bytes => 629145600, :memwd_max_strikes => 300, :memwd_rss_bytes => 1079683247, :message => "rss memory limit exceeded"} to include {:memwd_max_rss_bytes => 1258291200}

Lint

Each commit was checked on its own, in the image the docs-lint jobs use (lint-markdown:alpine-3.24-vale-3.21.0-markdownlint2-0.23.2-lychee-0.24.2-rumdl-0.2.69):

• markdownlint-cli2, scripts/lint-doc.sh and lychee --offline --include-fragments on the pages each commit changes: no markdownlint issues, lint-doc passed, no link errors.
• Vale at suggestion level, on each commit and on its parent: the same suggestions on both, none on the changed lines (six on the Sidekiq page, fifteen on the Puma page). The first commit moves the Sidekiq page's ReadingLevel from 8.67 to 8.63.
• Both commit messages pass scripts/lint/commit_linter.rb and the Danger commit linter (gitlab-dangerfiles 4.12.0).

Related issues

Closes #590922 (closed)

Author's checklist

• Optional. Consider taking the GitLab Technical Writing Fundamentals course.
• Follow the:
  • Documentation process.
  • Documentation guidelines.
  • Style Guide.
• If you're adding a new page, add the product availability details under the H1 topic title.
• If you are a GitLab team member, request a review based on:
  • The documentation page's metadata.
  • The associated Technical Writer.

If you are a GitLab team member and only adding documentation, do not add any of the following labels:

• ~"frontend"
• ~"backend"
• ~"type::bug"
• ~"database"

These labels cause the MR to be added to code verification QA issues.

Reviewer's checklist

Documentation-related MRs should be reviewed by a Technical Writer for a non-blocking review, based on Documentation Guidelines and the Style Guide.

If you aren't sure which tech writer to ask, use roulette or ask in the #docs Slack channel.

• If the content requires it, ensure the information is reviewed by a subject matter expert.
• Technical writer review items:
  • Ensure docs metadata is present and up-to-date.
  • Ensure the appropriate labels are added to this MR.
  • Ensure a release milestone is set.
  • If relevant to this MR, ensure content topic type principles are in use, including:
    • The headings should be something you'd do a Google search for. Instead of Default behavior, say something like Default behavior when you close an issue.
    • The headings (other than the page title) should be active. Instead of Configuring GDK, say something like Configure GDK.
    • Any task steps should be written as a numbered list.
    • If the content still needs to be edited for topic types, you can create a follow-up issue with the docs-technical-debt label.
• Review by assigned maintainer, who can always request/require the reviews above. Maintainer's review can occur before or after a technical writer review.