Increase pgbouncer "max_client_conn" for replica dbs
The pgbouncer instances for the production replica dbs is approaching its saturation point for the max number of connected clients. This is only relevant for the replicas, not the primary db. The pgbouncers have backend connections to spare, only the frontend max_client_conn is approaching saturation, so we can reasonably just increase that configured limit.
Background
Each of the replica dbs currently runs 3 pgbouncer instances, and each rails client connects to all of them.
The 7-day trend shows the connected client count regularly approaches the alerting threshold, and today it briefly exceeded it.
The graph below shows the 7-day trend for the metric underlying the above alert -- the number of rails clients connected to pgbouncer divided by pgbouncer's configured max client count.
In this graph, ignore yesterday's unrelated incident (severe lock contention over buffer-mapping lwlocks). The spike above the red line at the righthand side of the graph is what triggered today's alert.
Each of these pgbouncer instances has a high fan-in ratio of frontend to backend connections. Pgbouncer is able to satisfy its peak of 6K clients with less than 20 backend connections to postgres. And the trend is for 0 clients to be waiting for one of pgbouncer's backend connections to become available.
So increasing the max number of frontend client connections that each pgbouncer allows is a reasonable preventative action for this alert.
For reference, currently the ceiling is 9216 frontend client connections:
msmiley@patroni-v12-08-db-gprd.c.gitlab-production.internal:~$ sudo grep -r 'max_client_conn' /var/opt/gitlab/pgbouncer/
/var/opt/gitlab/pgbouncer/pgbouncer-2.ini:max_client_conn = 9216
/var/opt/gitlab/pgbouncer/pgbouncer.ini:max_client_conn = 9216
/var/opt/gitlab/pgbouncer/pgbouncer-1.ini:max_client_conn = 9216