Support opt-in sidekiq-cluster in bin/background_jobs script
What does this MR do?
gitlab-com/gl-infra/scalability#170
Part of the work required for gitlab-com/gl-infra&181 (closed).
This allows passing a SIDEKIQ_WORKERS to bin/background_scripts, if it's given, the script will use sidekiq-cluster under the hood (through the new bin/background_jobs_sk_cluster script) to spin the number of given workers. Otherwise, sidekiq will remain being used through bin/background_jobs_sk.
This script is currently being used by:
-
GDK though
bin/background_jobs start_foreground -
Source installations (
start,restartandstopsub-commands) - Deprecated sidekiq rake tasks
The goal is first making it available only at GDK (as opt-in) without affecting points 2 and 3.
In a later release, we'll roll it out for Source installations (gitlab-com/gl-infra/scalability#173) and remove the deprecated sidekiq.rake tasks (so no need to also use the script there - #196731 (moved))
It also simplifies the interface of bin/background_jobs as just:
-
start- Start on background -
restart- Kill all existing processes and start them again (in background). start_foregroundstop-
start_silent- A rename fromstart_no_daemonize(that should be removed at 13.0 - this is only used by the deprecatedsidekiq.rake)
Both killall and load_ok functions were removed as they're not being used anywhere.
When we finally stop using sidekiq directly, we can:
- Rename
bin/background_jobs_sk_clusterback tobin/background_jobs - Delete
bin/background_jobs_sk
Screenshots
Does this MR meet the acceptance criteria?
Conformity
-
Changelog entry -
Documentation (if required) -
Code review guidelines -
Merge request performance guidelines -
Style guides -
Database guides -
Separation of EE specific content
Availability and Testing
-
Review and add/update tests for this feature/bug. Consider all test levels. See the Test Planning Process. -
Tested in all supported browsers -
Informed Infrastructure department of a default or new setting change, if applicable per definition of done
Security
If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:
-
Label as security and @ mention @gitlab-com/gl-security/appsec -
The MR includes necessary changes to maintain consistency between UI, API, email, or other methods -
Security reports checked/validated by a reviewer from the AppSec team