# Request ## What kind of support are you looking for? - [ ] [Feature Group Stable Counterpart](https://handbook.gitlab.com/handbook/engineering/data-engineering/database-excellence/database-frameworks/stable/) - [ ] Dedicated team member/resource - [x] Advice on structure/performance/etc ## What team? **Handbook:** https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/tenant-scale/tenant-services/ **Slack Channel:** #g_tenant_services **Engineering Manager:** @jarv **Label:** `~"group::tenant services"` ## Describe the feature or ongoing work that needs assistance As part of [&56 — Isolate Sidekiq workloads to reduce blast radius](https://gitlab.com/groups/gitlab-com/gl-infra/tenant-scale/tenant-services/-/work_items/56), we are working to improve isolation of the Sidekiq quarantine shard. The quarantine shard segregates problematic workers (e.g. `AuditEvents::AuditEventStreamingWorker`, `Members::DestroyWorker`) from the main Sidekiq fleet. **Current situation:** The quarantine shard connects to the same PgBouncer pools as all non-urgent shards: | PgBouncer host | Pool | Pool size | Backend | |----------------|------|-----------|---------| | `pgbouncer-sidekiq` | `gitlabhq_production_sidekiq` | 25 | patroni-main | | `pgbouncer-sidekiq-ci` | `gitlabhq_production_sidekiq` | 30 | patroni-ci | | `pgbouncer-sidekiq-sec` | `gitlabhq_production_sidekiq` | 30 | patroni-sec | A misbehaving worker on quarantine that saturates the PgBouncer pool (long-running queries, idle-in-transaction connections) will starve **all** non-urgent Sidekiq shards of database connections, The opposite of what quarantine is meant to achieve (INC-10096, INC-10084). **What we are proposing:** Create a separate PgBouncer pool `gitlabhq_production_sidekiq_quarantine` on each of the three `pgbouncer-sidekiq` fleets with a small, dedicated pool size, similar to how `gitlabhq_production_sidekiq_urgent` gives urgent shards their own pool. ## What we want advice on 1. Is a dedicated pool the right mechanism, or are there better alternatives (e.g. per-role connection limits, statement timeouts, idle-in-transaction timeouts on the quarantine user)? 2. Pool sizing recommendation: what is an appropriate pool size for a bursty, low-urgency worker like `AuditEvents::AuditEventStreamingWorker`? 3. Any risks we should be aware of with a very small dedicated pool (e.g. connection acquisition timeouts causing job failures rather than queuing)? 4. Should this change be made before or after provisioning a dedicated Redis instance for the quarantine queue? ## Expectations for participating member(s) of the database group in the target group/project - Async review of our proposal (this issue + a follow-up implementation MR once we agree on approach) - One sync session if needed to agree on pool sizing and approach - Expected SLO: initial response within 1 week ## Expectations for participating member(s) of the database group in the database frameworks group - Advisory engagement only : No ongoing ownership expected - No on-call or emergency implications - Estimated effort: 2–4 hours total (review + one sync if needed) ## Exit criteria - [ ] Decision documented: dedicated pool (with agreed sizing) or alternative approach - [ ] If proceeding with a dedicated pool: Tenant Services can implement independently based on agreed parameters - [ ] Implementation MR reviewed by DBRE counterpart before merge ## Checklist ### Requesting Team - [x] The issue has a descriptive title - [x] There are detailed answers to the questions above - [ ] The issue is assigned to the [database team manager](https://about.gitlab.com/handbook/engineering/development/enablement/database/#team-members) - [ ] If this is urgent, reach out to the team manager in Slack ### Database Team - [ ] There is enough information to prioritize the request - [ ] The request has been assigned to a member of the team - [ ] The priority of the request has been agreed by the stakeholders and author