Investigate reducing shared_buffers for PostgreSQL
This value is currently set to 112896MB, which is based on our old setup of using a single DB for everything. Now that we're using DB load balancing and pgbouncer we should re-investigate this value and lower it if this is beneficial.
We need to be able to quickly restart the primary. This requires that we increase the repmgr check interval so it doesn't promote a new primary during this restart. We must also first stop Sidekiq to reduce pressure on the primary, allowing it to restart faster.
A failover won't help, as IIRC the value for this setting on secondaries must be equal or greater than the primary. If this is not the case the secondary will refuse to start, making it impossible to promote it to a primary. One hack would be to edit the configuration file manually, promote it, then restart it. However, if Chef notices the file is different in the mean time it may revert this change and restart the secondary unexpectedly. As such I think simply restarting the primary is more reliable.