Direct more Sidekiq queries to replicas where possible
Now that we have database loadbalancing available for Sidekiq, we should see if there's more workers that could be directing queries to replicas instead of the primary.
This would reduce the number of database connections for the primary needed, a resource that's currently close to saturation (https://gitlab.com/gitlab-com/gl-infra/capacity-planning/-/issues/32) and we can't easily increase (https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/15628).
Some workers to look into as noted by Andrew (https://gitlab.com/gitlab-com/gl-infra/capacity-planning/-/issues/32#note_901279760):
This is how things look over the past 7 days.
-
RepositoryUpdateMirrorWorker
is the most expensive Sidekiq worker: This worker is scheduled fromProjectImportScheduleWorker
and requires the database changes by that worker. If we want to direct reads to a replica, we'll have to introduce some scheduling delay. This might be acceptable. -
ProjectImportScheduleWorker
may also be a good candidate for investigation. This worker loads a project with a bunch of relationships and performs a write to mark the mirror as in-progress. Could this usedata_consistency :sticky
?
https://log.gprd.gitlab.net/goto/9cd02150-b4ec-11ec-b73f-692cc1ae8214