Set up pgbouncer
Prior testing done here: https://gitlab.com/gitlab-com/infrastructure/issues/259#note_25131536
The plan for moving to pgbouncer is as follows:
- Install via apt-get on db3
- Configure 1000 frontend connections, 200 max_connections for PostgreSQL
- Set up authentication so it uses PostgreSQL's own credentials
- Swap the ports of PostgreSQL and pgbouncer
- Monitor
Once this works we can apply it to db4 as well. We will then monitor/use this for a day or so to see how it behaves when both secondaries use pgbouncer. Once this works we need to take the following steps:
- Move pgbouncer back to a separate port instead of using PostgreSQL's port
- Move the secondary ports back to the default
- Set up pgbouncer on the primary
- Configure GitLab to use the pgbouncer port instead of the PostgreSQL port
- Drop max_connections on the primary to 200 (to match the secondaries)
- Restart PostgreSQL
In theory the last step does not require downtime as IIRC pgbouncer can handle restarts, but since this is the first time we'll be doing this I suggest enabling the deploy page to be certain.
Once this is working we can move towards using pgbouncer as shipped with omnibus (gitlab-org/omnibus-gitlab!1345 (merged) still a WIP).