Load balancing for Sidekiq
Currently Sidekiq always uses the primary. In theory we should be able to also use the secondaries. The basic idea of achieving this is as follows: get the WAL position when scheduling a job, then include this as job metadata. A worker will then check this to see if it can start off with a secondary.
However, before we implement this we should investigate how beneficial this would be. To do so, we need to take these steps:
- Modify the load balancer so it's enabled for Sidekiq, but forces all queries to always go to the primary
- Instrument the methods used for reads/writes so we can count (using Prometheus):
- The number of reads from a secondary
- The number of writes
- The number of reads from the primary
- Deploy this as a temporary patch to a few Sidekiq workers on GitLab.com
- Let this run for a good 15 minutes or so to get data
- Revert, then investigate the data