Service discovery for database load balancing
Instead of manually configuring all secondaries to balance load amongst, we need a service discovery system. This system works basically as follows:
1. Look up a DNS record for the primary. This record will contain the host to connect to for writes.
1. Look up a DNS record for all secondaries. This record will return multiple values, each specifying a secondary to connect to.
1. Configure ActiveRecord using the above settings
To handle refreshing we'll have a thread re-query this data every N minutes (not sure yet what a good default would be), with a small random value added to spread load a bit. If this thread notices the list of hosts is different it will:
1. Reconfigure ActiveRecord using the new hosts
1. Disconnect all existing connections
A hard disconnect might result in errors, or result in a request still trying to use old hosts (e.g. by immediately trying to reconnect). As such we probably need some kind of barrier/safe point before reconnecting.
This also means having to change the configuration format. This should be clearly documented as users will need to change `database.yml` accordingly for otherwise load balancing will be disabled.
issue