In https://gitlab.com/groups/gitlab-org/-/epics/5592, within the scope of the rapid action, we focused on enabling Sidekiq to use read-only database replicas capabilities and to allow BuildHooksWorker worker to utilize these new capabilities. The broader goal is to allow Sidekiq to use read replicas so that read-only jobs don't put pressure on the primary Postgres instance. It should be relatively easy to migrate other read-only workers once the pattern has been established. We should also focus on other discussed improvements such as: - Better retry mechanism for :delayed data consistency - Provide correct `database_replica_location` to the worker middleware - Allow idempotent sidekiq jobs to use replicas as well - Cache database log location on the host level, since we are scheduling jobs in batches. ## Exit criteria - [X] Improve documentation and create Show&Tell for Sidekiq Load Balancing to allow other teams to convert more workers to utilize LB - We now have [Slides](https://slides.com/nikolamilojevic/deck-eb7ff8/fullscreen), [Show and Tell](https://youtu.be/PAJ8nvG1AGA) and updated [documentation](https://docs.gitlab.com/ee/development/sidekiq_style_guide.html#job-data-consistency-strategies) - [X] Move several workers to utilize LB (To measure the impact and as an example for other teams) - [X] Improve our metrics so we can have a better picture of our load balancing strategy/decision making - [X] Provide a way to measure the overall impact of switching traffic from the primary to the replicas - We can monitor impact with our [pg-internal metric](https://gitlab.com/groups/gitlab-org/-/epics/5691#note_600757973) - [X] Remove 10% of transactions from hitting the primary database - https://gitlab.com/groups/gitlab-org/-/epics/5691#note_606250739 - [x] Enforce decision on utilizing LoadBalancing for the new workers - https://gitlab.com/gitlab-org/gitlab/-/issues/334495 - [x] Enable LB for Sidekiq by default - https://gitlab.com/gitlab-org/gitlab/-/issues/334494 - [x] Close all issues already in development (scheduled for 14.1) Optional: - [x] Add support for idempotent workers and deduplication - https://gitlab.com/gitlab-org/gitlab/-/issues/325291 - [x] Create a rubocop rule when enabling LB for idempotent jobs