Use HAProxy in front of secondaries to handle failure case

From this comment by @northrup

After talking with @yorickpeterse the architectural model that I'll be adopting is one of sidecar, where each DB node will run a thin HA-Proxy that sits in front of the database and is responsible for abstracting postgres failure. Primary communications to postgres will be on the standard port, secondary communications to postgres will be on a non-standard port, both of which will be answered by HA-Proxy.

For the database primary roll, CRM will initiate the script that changes the firewall state which envokes relocation of the IP address associated with the Primary Database. For the secondary status we will be running an HA-Proxy service check on the local host to ensure that our local postgres server is reachable. Each database server will select itself for all secondary queries presented to it and ONLY reference another secondary server in the event that the local server is unavailable.


Original description: Currently we only have an Azure load balancer for the primary. For database load balancing to work properly in the event of a failover we also need Azure load balancers for every secondary. During a failover this means we update 2 Azure load balancers (one pointing to the new primary, and one pointing to the previous-primary-now-secondary), and we're good to go.