Skip to content

Enable load balancing for PipelineHooksWorker [RUN ALL RSPEC] [RUN AS-IF-FOSS]

Matthias Käppler requested to merge 324375-pipeline-hooks-worker-lb into master

What does this MR do?

We recently introduced the ability to use database read replicas in sidekiq workers, which relieves pressure on the primary. This is possible as long as workers do not issue any writes. This worker has been identified as a candidate; we found that it only writes to the primary in 0.0008% of all job executions in any given week. There is therefore a lot of potential to move millions of executions over to read from replicas instead.

Using delayed consistency means that:

  • New jobs will be enqueued with a 1 second delay to reduce the chance that we encounter replication lag
  • We will read from replicas unless they are lagging behind the primary
  • If replication lag is encountered, we retry the job

The change is behind the load_balancing_for_pipeline_hooks_worker feature toggle: #331424 (closed)

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Related to #324375 (closed)

Edited by Matthias Käppler

Merge request reports