Use the dedicated pgbouncer cluster for production

Production Change - Criticality 1 C1

Related to #895 (closed)

Change Objective	Describe the objective of the change
Change Type	Type described above
Services Impacted	~"Service:pgbouncer" ~"Service:Sidekiq" ~"Service:Web" ~"Service:API" ~"Service:Git"
Change Team Members	@aamarsanaa
Change Severity	How critical is the change
Buddy check	@ahmadsherif @abrandl
Tested in staging	The change was tested on staging environment
Schedule of the change	Date and time (with timezone)
Duration of the change	Time to execute the change ( including a possible rollback )
Downtime Component	if yes how many minutes
Detailed steps for the change. Each step must include:	See below

Steps, Part 1

1. Stop chef-client on all clients: knife ssh 'roles:gprd-base-fe OR roles:gprd-base-be' "sudo service chef-client stop"
2. Merge and apply https://ops.gitlab.net/gitlab-cookbooks/chef-repo/merge_requests/1248
3. Apply the change manually on one client, make sure it's serving requests properly
4. Roll-out the change gradually: e.g. knife ssh -C 2 'roles:gprd-base-fe OR roles:gprd-base-be' "sudo chef-client"

Rollback

1. Revert and apply the merge request

Steps, Part 2 (optional)

This step is only needed if we need two pgbouncer nodes to serve the clients

1. Stop chef-client on all pgbouncer nodes: knife ssh 'roles:gprd-base-db-pgbouncer' "sudo service chef-client stop"
2. Merge and apply https://ops.gitlab.net/gitlab-cookbooks/chef-repo/merge_requests/1249
3. Run chef-client on the current pgbouncer leader
4. Run chef-client on the rest of the nodes

Rollback

1. Revert and apply the merge request
2. HUP and/or restart clients to force them to drop connections to the no-longer-be-used pgbouncer node