Skip to content

Support opt-in sidekiq-cluster in bin/background_jobs script

What does this MR do?

gitlab-com/gl-infra/scalability#170

Part of the work required for gitlab-com/gl-infra&181 (closed).

This allows passing a SIDEKIQ_WORKERS to bin/background_scripts, if it's given, the script will use sidekiq-cluster under the hood (through the new bin/background_jobs_sk_cluster script) to spin the number of given workers. Otherwise, sidekiq will remain being used through bin/background_jobs_sk.

This script is currently being used by:

  1. GDK though bin/background_jobs start_foreground
  2. Source installations (start, restart and stop sub-commands)
  3. Deprecated sidekiq rake tasks

The goal is first making it available only at GDK (as opt-in) without affecting points 2 and 3.

In a later release, we'll roll it out for Source installations (gitlab-com/gl-infra/scalability#173) and remove the deprecated sidekiq.rake tasks (so no need to also use the script there - #196731 (moved))

It also simplifies the interface of bin/background_jobs as just:

  1. start - Start on background
  2. restart - Kill all existing processes and start them again (in background).
  3. start_foreground
  4. stop
  5. start_silent - A rename from start_no_daemonize (that should be removed at 13.0 - this is only used by the deprecated sidekiq.rake)

Both killall and load_ok functions were removed as they're not being used anywhere.

When we finally stop using sidekiq directly, we can:

  1. Rename bin/background_jobs_sk_cluster back to bin/background_jobs
  2. Delete bin/background_jobs_sk

Screenshots

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

  • Label as security and @ mention @gitlab-com/gl-security/appsec
  • The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
  • Security reports checked/validated by a reviewer from the AppSec team
Edited by Oswaldo Ferreira

Merge request reports

Loading