Investigate ulimits for pgbouncer
During a recent failover production#637 (closed), we had an issue where when the new leader had been elected, this node still had pgbouncer connections around from when it was a secondary. Due to this, new connections were unable to be made to this server as we had hit the ulimit 4096.
To remediate this we had to forcibly modify chef to remove node-04 from the pool and run chef to restart our ruby applications.
Consider bumping up file descriptor limitations to prevent this issue from happening should we have another failover.
Related/Duplicated: