Allow config Sidekiq worker memory limit in ENV
What does this MR do?
For #32212 (closed)
Sidekiq daemon memory killer allow whitelist
worker memory usage when checking the Sidekiq worker RSS.
To be whitelisted, the worker need to set sidekiq options memory_killer_memory_growth_kb
and memory_killer_max_memory_growth_kb
.
Per #32212 (comment 218637644), GitLab.com expects to config these sidekiq_options, and environment variable is an acceptable option.
This MR starts from 3 identified workers(for project import) and set their sidekiq options memory limit, reading the environment variables.
The 3 workers are: app/workers/gitlab/github_import/stage/finish_import_worker.rb, app/workers/gitlab/github_import/stage/import_repository_worker.rb, app/workers/repository_import_worker.rb
The memory limit ENV name follow a convention of: MEMORY_KILLER_#{worker_name}_MEMORY_GROWTH_KB
, and MEMORY_KILLER_#{worker_name}_MAX_MEMORY_GROWTH_KB
For example, for worker app/workers/gitlab/github_import/stage/finish_import_worker.rb
, the options are set as:
sidekiq_options memory_killer_memory_growth_kb: ENV.fetch('MEMORY_KILLER_FINISH_IMPORT_WORKER_MEMORY_GROWTH_KB', 50).to_i
sidekiq_options memory_killer_max_memory_growth_kb: ENV.fetch('MEMORY_KILLER_FINISH_IMPORT_WORKER_MAX_MEMORY_GROWTH_KB', 200_000).to_i
The default value 50
is estimated in GDK environment.