Service discovery for database load balancing
Instead of manually configuring all secondaries to balance load amongst, we need a service discovery system. This system works basically as follows:
- Look up a DNS record for the primary. This record will contain the host to connect to for writes.
- Look up a DNS record for all secondaries. This record will return multiple values, each specifying a secondary to connect to.
- Configure ActiveRecord using the above settings
To handle refreshing we'll have a thread re-query this data every N minutes (not sure yet what a good default would be), with a small random value added to spread load a bit. If this thread notices the list of hosts is different it will:
- Reconfigure ActiveRecord using the new hosts
- Disconnect all existing connections
A hard disconnect might result in errors, or result in a request still trying to use old hosts (e.g. by immediately trying to reconnect). As such we probably need some kind of barrier/safe point before reconnecting.
This also means having to change the configuration format. This should be clearly documented as users will need to change database.yml accordingly for otherwise load balancing will be disabled.