Service discovery for database load balancing
Instead of manually configuring all secondaries to balance load amongst, we need a service discovery system. This system works basically as follows: 1. Look up a DNS record for the primary. This record will contain the host to connect to for writes. 1. Look up a DNS record for all secondaries. This record will return multiple values, each specifying a secondary to connect to. 1. Configure ActiveRecord using the above settings To handle refreshing we'll have a thread re-query this data every N minutes (not sure yet what a good default would be), with a small random value added to spread load a bit. If this thread notices the list of hosts is different it will: 1. Reconfigure ActiveRecord using the new hosts 1. Disconnect all existing connections A hard disconnect might result in errors, or result in a request still trying to use old hosts (e.g. by immediately trying to reconnect). As such we probably need some kind of barrier/safe point before reconnecting. This also means having to change the configuration format. This should be clearly documented as users will need to change `database.yml` accordingly for otherwise load balancing will be disabled.
issue