Direct reads from Sidekiq to read-only replicas by default
For web requests, any read database query will be directed at a read-only replica if one can be found that is sufficiently caught up to the primary.
In gitlab-org/gitlab#322452 (closed) we've built a
similar mechanism for Sidekiq, but specific workers can opt in to
using load balancing by specifying a
data_consistency
worker attribute. The default data_consistency
is set to always
,
meaning that a worker will always use the primary for all queries.
To prevent stale reads, we store the WAL of all primary databases in Redis the moment we schedule a job. When the job gets picked up, we'll use that wal information to determine if a replica has sufficiently caught up.
Using this, we should be able to use the same kind of loadbalancing strategy for Sidekiq by default: Jobs that start prefer using a replica, when they perform a write, they will start sticking to the primary.