Support opt-in sidekiq-cluster in bin/background_jobs script
What does this MR do?
gitlab-com/gl-infra/scalability#170
Part of the work required for gitlab-com/gl-infra&181 (closed).
This allows passing a SIDEKIQ_WORKERS
to bin/background_scripts
, if it's given, the script will use sidekiq-cluster
under the hood (through the new bin/background_jobs_sk_cluster
script) to spin the number of given workers. Otherwise, sidekiq
will remain being used through bin/background_jobs_sk
.
This script is currently being used by:
-
GDK though
bin/background_jobs start_foreground
-
Source installations (
start
,restart
andstop
sub-commands) - Deprecated sidekiq rake tasks
The goal is first making it available only at GDK (as opt-in) without affecting points 2
and 3
.
In a later release, we'll roll it out for Source installations (gitlab-com/gl-infra/scalability#173) and remove the deprecated sidekiq.rake
tasks (so no need to also use the script there - #196731 (moved))
It also simplifies the interface of bin/background_jobs
as just:
-
start
- Start on background -
restart
- Kill all existing processes and start them again (in background). start_foreground
stop
-
start_silent
- A rename fromstart_no_daemonize
(that should be removed at 13.0 - this is only used by the deprecatedsidekiq.rake
)
Both killall
and load_ok
functions were removed as they're not being used anywhere.
When we finally stop using sidekiq
directly, we can:
- Rename
bin/background_jobs_sk_cluster
back tobin/background_jobs
- Delete
bin/background_jobs_sk
Screenshots
Does this MR meet the acceptance criteria?
Conformity
-
Changelog entry -
Documentation (if required) -
Code review guidelines -
Merge request performance guidelines -
Style guides -
Database guides -
Separation of EE specific content
Availability and Testing
-
Review and add/update tests for this feature/bug. Consider all test levels. See the Test Planning Process. -
Tested in all supported browsers -
Informed Infrastructure department of a default or new setting change, if applicable per definition of done
Security
If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:
-
Label as security and @ mention @gitlab-com/gl-security/appsec
-
The MR includes necessary changes to maintain consistency between UI, API, email, or other methods -
Security reports checked/validated by a reviewer from the AppSec team