Fix Client Connections saturating on primary PGBouncer
Summary
We've been violating our SLO for pgbouncer_client_conn_primary component
since we are saturating it, and sometimes hitting 85% saturation.
In the past 2 weeks we violated this SLO 9 times:
Related Incident(s)
Originating issue(s):
Desired Outcome/Acceptance Criteria
-
Increase our saturation SLO to be 95% so we don't page on-call with non actionable items. 👉 gitlab-com/runbooks!4862 (merged) -
We need to test increasing the client limit on each pgbouncer because pgbouncer is a single thread process, therefore we don’t know if having more client per process can cause any performance impact… it would be safer to add add an extra pgbouncer port per main pgbouncer node to handle extra 8192 clients, but this will also open additional server connections 👉 production#7536 (closed) -
Revert the modification for the SLO to page again on 85% 👉 gitlab-com/runbooks!4873 (merged)
Associated Services
Corrective Action Issue Checklist
-
Link the incident(s) this corrective action arose out of -
Give context for what problem this corrective action is trying to prevent from re-occurring -
Assign a severity label (this is the highest sev of related incidents, defaults to 'severity::4') -
Assign a priority (this will default to 'priority::4')
Edited by Steve Xuereb