## Production Change - Criticality 1 ~"C1" | Change Objective | PgBouncer endpoint addition (and migration) for sidekiq as a frontend pool for RW | |:-------------------------------------------------------|:-------------------------------------------------------------------------------------------------------------------------------------------------| | Change Type | Frontend PgBouncers Pool split (Sidekiq and Production) | | Services Impacted | ~"Service:Postgres" ~"Service:Pgbouncer" | | Change Team Members | SRE - Ongres/ @hphilipps / @ahmadsherif / @emanuel_ongres | | Change Severity | ~C1 | | Buddy check or tested in staging | @ahmadsherif @emanuel_ongres @gerardo.herzig | | Schedule of the change | 12 of December of 2019 00:00 utc| | Duration of the change | 1 hour | ## General Due to PgBouncer limitations regarding its single-core architecture and, the inability to manage pools in an isolated manner (that is, all pools will in the same queue), it is necessary to split both pool families (production and sidekiq) into their own dedicated resources. Due to the criticality of this issue and considering that it is needed to keep cookbooks clean and agnostic (See [PgBouncer CookBooks](https://gitlab.com/gitlab-cookbooks/gitlab-pgbouncer)), it is preferred to spawn a different set of parallel resources to the current PgBouncer Layer. Within this, we are also increasing the client connection capacity for both Sidekiq and Production queues, by keeping the `max_client_conn=4096`. By having services split, `min_pool_size` can be set on both PgBouncer clusters, which previously impacted due that it affects each defined pool. Now, it is possible to set different sizes of minimum persistent connections for each pool. The infrastructure will end up having 2 PgBouncer layers against the current leader: ```mermaid graph TB pg3S --> leader pg2S --> leader pg1S -.-> leader pg3 --> leader pg2 --> leader pg1 -.-> leader subgraph SLB["Sidekiq iLB"] SiLB["Production iLB"] -.-> pg1["PgBouncer1"] SiLB["Production iLB"] --> pg2["PgBouncer2"] SiLB["Production iLB"] --> pg3["PgBouncer3"] end subgraph PLB["Production iLB"] PrLB["Sidekiq iLB"] -.-> pg1S["PgBouncer1"] PrLB["Sidekiq iLB"] --> pg2S["PgBouncer2"] PrLB["Sidekiq iLB"] --> pg3S["PgBouncer3"] end subgraph PG["Patroni"] leader replica replicaN end ``` Some other related facts around this CR: Originally, this Change Request scoped a PgBouncer service addition which considered an extension of the [PgBouncer CookBook]. Although, after a review between SRE Team and OnGres, it was resolved that adding different roles in the same cookbook will not be convenient in a time-wise manner and code maintainability. Also, this issue included a switch of the pool size values between Production and Sidekiq queue, which has been already covered at https://gitlab.com/gitlab-com/gl-infra/production/issues/1373. There is no need for failover in order to execute the following change, as RW pool resides in a different layer. Although, some additional operation steps should be reviewed and executed. ## Motivation The motivation under this re-architecture of the frontend PgBouncer is to: - Avoid incoming large-operations from sidekiq to impact production operations - Allow deploying different pool sizes for each pool depending on the accumulated operations in sidekiq. That is, controlling the maximum active connections from sidekiq, in order to avoid unnecessary concurrency. - It will enhance the observability of each pool behavior. ## Plan @ansdval for IMOC awareness. @cmiskell please have a look at the tasks that we need your help, thanks. - [x] Stop `chef-client` on the original cluster [@cmiskell] - [x] Apply the new database configuration (adjusted `pool_size` and whatnot) to Chef, *don't* roll it to any nodes yet ([MR](https://ops.gitlab.net/gitlab-cookbooks/chef-repo/merge_requests/2296)) [@finotto] - [x] Provision the new cluster in [terraform](https://ops.gitlab.net/gitlab-com/gitlab-com-infrastructure/merge_requests/1233) [@cmiskell] - [x] Make sure that pgbouncer can establish connections to primary database [@emanuel_ongres] - [x] Stop `chef-client` on all Sidekiq nodes [@cmiskell] - [x] Apply the [change](https://ops.gitlab.net/gitlab-cookbooks/chef-repo/merge_requests/2297) that points Sidekiq to the new cluster [@Finotto ] - [x] One by one, run `chef-client` on the Sidekiq nodes, wait/sleep for the value of `server_idle_timeout` (currently `65`) or a little higher, then move on to the next node [@cmiskell] - [x] On the inactive pgbouncer from the original cluster, run `chef-client` to apply the database configuration. - [x] Through Terrafrom, change the health check from HTTP over 8010 to TCP over 6432 ([MR](https://ops.gitlab.net/gitlab-com/gitlab-com-infrastructure/merge_requests/1224)) [@Finotto] - [x] Re-enable `chef-client` on the original cluster [@cmiskell] - [x] Across both clusters, stop and disable `pgbouncer-leader-check` service as it's no longer needed, remove `/etc/systemd/system/pgbouncer-leader-check.service` and do a `systemctl daemon-reload` [@cmiskell] (*) Health Check is done against `6432` TCP **rollback :** - revert changes: - [ ] Revert the new database configuration (adjusted `pool_size` and whatnot) to Chef, *don't* roll it to any nodes yet ([MR](https://ops.gitlab.net/gitlab-cookbooks/chef-repo/merge_requests/2296)) [@finotto] - [ ] Apply chef changes on the original pgbouncer nodes. - [ ] Revert the [change](https://ops.gitlab.net/gitlab-cookbooks/chef-repo/merge_requests/2297) that points Sidekiq to the new cluster [@Finotto ] - [ ] One by one, run `chef-client` on the Sidekiq nodes, wait/sleep for the value of `server_idle_timeout` (currently `65`) or a little higher, then move on to the next node Prepare to monitor before traffic starts to flow in (OnGres): - [x] Get dashboards from the new iLB (StackDriver). - [x] Connect to all the available PgBouncers in order to monitor the traffic holistically. At this point, iLB should be pointing to PgBouncer Sidekiq Instance Groups, within Health Check against TCP 6432 in place and applied. - [x] Wait until PgBouncer Production pools drain sidekiq traffic. - OnGres @gerardo.herzig: please collect here the links of the necessary dashboards or scripts to do so. ### Finalize * [x] Monitor that traffic flows as usual. ## Changes -- Updated through https://gitlab.com/gitlab-com/gl-infra/production/issues/1373 Additionally, it is important to highlight the present imbalance between the sum between the production and sidekiq pools. From the overall maximum of 250 active connections for all _active_ PgBouncers: - 150 are for Production - 100 are for Sidekiq This configuration leads towards potential large operations (usually coming from Sidekiq) under heavy concurrency. It is recommended to switch the pool size (that is, the active connections), ending up with 150 (75) for production and, 100 for sidekiq (50 each). MRs: pgbouncer roles : https://ops.gitlab.net/gitlab-cookbooks/chef-repo/merge_requests/2296 changing sidekiq pgbouncer node: https://ops.gitlab.net/gitlab-cookbooks/chef-repo/merge_requests/2297 MR for change the healthcheck port on terraform : https://ops.gitlab.net/gitlab-com/gitlab-com-infrastructure/merge_requests/1224 Mr to remove the ha-check from pgbouncer : https://ops.gitlab.net/gitlab-cookbooks/chef-repo/merge_requests/2298 [PgBouncer CookBook]: https://gitlab.com/gitlab-cookbooks/gitlab-pgbouncer