Skip to content

Allow config Sidekiq worker memory limit in ENV

Qingyu Zhao requested to merge read-sidekiq-worker-memory-limit-from-env into master

What does this MR do?

For #32212 (closed)

Sidekiq daemon memory killer allow whitelist worker memory usage when checking the Sidekiq worker RSS. To be whitelisted, the worker need to set sidekiq options memory_killer_memory_growth_kb and memory_killer_max_memory_growth_kb.

Per #32212 (comment 218637644), GitLab.com expects to config these sidekiq_options, and environment variable is an acceptable option.

This MR starts from 3 identified workers(for project import) and set their sidekiq options memory limit, reading the environment variables.

The 3 workers are: app/workers/gitlab/github_import/stage/finish_import_worker.rb, app/workers/gitlab/github_import/stage/import_repository_worker.rb, app/workers/repository_import_worker.rb

The memory limit ENV name follow a convention of: MEMORY_KILLER_#{worker_name}_MEMORY_GROWTH_KB, and MEMORY_KILLER_#{worker_name}_MAX_MEMORY_GROWTH_KB

For example, for worker app/workers/gitlab/github_import/stage/finish_import_worker.rb, the options are set as:

        sidekiq_options memory_killer_memory_growth_kb: ENV.fetch('MEMORY_KILLER_FINISH_IMPORT_WORKER_MEMORY_GROWTH_KB', 50).to_i
        sidekiq_options memory_killer_max_memory_growth_kb: ENV.fetch('MEMORY_KILLER_FINISH_IMPORT_WORKER_MAX_MEMORY_GROWTH_KB', 200_000).to_i

The default value 50 is estimated in GDK environment.

Merge request reports