Skip to content

Allow Sidekiq jobs to use readonly database replicas

Nikola Milojevic requested to merge 322452-sidekiq-use-replicas into master

What does this MR do?

  1. This MR allows us to enable LoadBalancing for the Sidekiq by setting the ENV variable ENABLE_LOAD_BALANCING_FOR_SIDEKIQ to 'true'

  2. It allows us to define the data consistency requirement for a worker:

  • always: the worker is required to use primary (a default)
  • sticky: worker would use replica as long as possible, but would switch to primary either on write or long replication lag: use on jobs that require to be executed as fast as possible
  • delayed: worker would switch to primary only on write, would use replica always if there's a long replication lag the job will be delayed, and only if the replica is not up to date on the next retry, it will switch to the primary. It should be used on jobs where we are fine to delay the execution of a given job, due to their importance: expire caches, or execute hooks...

It is also possible to control data consistency configuration with the feature flag for each worker:

data_consistency :delayed, feature_flag: load_balancing_for_build_hooks_worker

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

  • Label as security and @ mention @gitlab-com/gl-security/appsec
  • The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
  • Security reports checked/validated by a reviewer from the AppSec team

Related to #322452 (closed)

Edited by Nikola Milojevic

Merge request reports