Partial synchronous replication to balance primary load

This is an idea I had earlier today, and while perhaps not useful it's worth getting some feedback on.

Currently the primary is used by Sidekiq for boths reads and writes, and by Unicorn for reads that take place after a write. In both cases this means the primary is still performing quite a bit of work. We could spread the load a bit more by introducing an additional secondary that replicas synchronously from the primary. This would lead to the following setup:

• db1 (primary)
• db3 (async secondary)
• db4 (async secondary)
• db5 (sync secondary)

We then need to adjust DB load balancing so that the host used for reads taking place after a write is db5, instead of the primary (db1).

In this setup the primary is only used for writes, and db5 is used for regular reads and for reads taking place after a write. Because this host is always in sync with the primary we can also use this for Sidekiq, reducing the primary load further.

Downsides of this approach:

• Somewhat ironically writes will be a bit slower because a COMMIT has to be completed (and visible) on 2 hosts instead of 1
• We need to adjust database load balancing so it can fall back to this sync host instead of the primary (while still falling back to the primary if nothing else is online)
• The primary CPU usage hardly ever crosses 20%, so one could argue this setup is a bit of an overkill
• We need to adjust PostgreSQL's configuration so a COMMIT will not return until it is visible on the sync host
• The more sync hosts we add, the slower writes will be because IIRC they don't happen in parallel; instead they are performed 1 by 1

cc @pcarranza