Load balancing for Sidekiq

Currently Sidekiq always uses the primary. In theory we should be able to also use the secondaries. The basic idea of achieving this is as follows: get the WAL position when scheduling a job, then include this as job metadata. A worker will then check this to see if it can start off with a secondary.

However, before we implement this we should investigate how beneficial this would be. To do so, we need to take these steps:

  1. Modify the load balancer so it's enabled for Sidekiq, but forces all queries to always go to the primary
  2. Instrument the methods used for reads/writes so we can count (using Prometheus):
  3. The number of reads from a secondary
  4. The number of writes
  5. The number of reads from the primary
  6. Deploy this as a temporary patch to a few Sidekiq workers on GitLab.com
  7. Let this run for a good 15 minutes or so to get data
  8. Revert, then investigate the data