## :dart: Opportunity: Sec GPRD Production Cluster Compute and Disk Right-Sizing Optimization As part of ongoing infrastructure cost optimization, we've identified a right-sizing opportunity in the **Patroni Sec Cluster (Production / GPRD)**. Following the PG18 upgrade, the cluster was migrated from `n2-highmem-96` to `c4-highmem-96` nodes, but the compute profile has not yet been re-evaluated against the actual post-upgrade load. Additionally, the data disk shows meaningful headroom that warrants review. This issue tracks the analysis, planning, and implementation of compute and disk right-sizing for this cluster. > :warning: **Log disk right-sizing is explicitly out of scope for this issue.** In production, we do not want to risk running out of log disk space if Patroni or PostgreSQL generates excessive logs before the cleanup mechanism takes place. Log disk sizing will not be changed. This follows the same procedure established for the Patroni Sec Cluster optimization ([dbo-issue-tracker#510](https://gitlab.com/gitlab-com/gl-infra/data-access/dbo/dbo-issue-tracker/-/work_items/510)) and the Staging Registry Cluster work ([dbo-issue-tracker#751](https://gitlab.com/gitlab-com/gl-infra/data-access/dbo/dbo-issue-tracker/-/work_items/751)). CRs and procedures from those issues can be referenced and adapted. --- ### :bar_chart: Current Cluster Configuration **Cluster:** `gprd-patroni-sec-v18` | Member | Host | IP | Role | State | TL | Lag in MB | Tags | |--------|------|----|------|-------|----|-----------|------| | patroni-sec-v18-101-db-gprd.c.gitlab-production.internal | 10.220.41.201 | Leader | running | 2 | | | | patroni-sec-v18-102-db-gprd.c.gitlab-production.internal | 10.220.41.202 | Replica | streaming | 2 | 18 | backup_node: true, console_node: true, nofailover: true, noloadbalance: true | | patroni-sec-v18-103-db-gprd.c.gitlab-production.internal | 10.220.41.203 | Replica | streaming | 2 | 0 | | | patroni-sec-v18-104-db-gprd.c.gitlab-production.internal | 10.220.41.204 | Replica | streaming | 2 | 0 | | **Total: 4 nodes (1 Leader + 3 Replicas)** **Current VM type (all nodes):** `c4-highmem-96` (96 vCPUs, 744 GB Memory) **Previous VM type (pre-PG18 upgrade):** `n2-highmem-96` (96 vCPUs, 768 GB Memory) > :bulb: The backup node (`patroni-sec-v18-102`) is also running `c4-highmem-96` and is considered **oversized** given its role (backup + console, nofailover, noloadbalance). It is a strong candidate for independent right-sizing. --- ### :package: Current Storage Utilization | Mount | Device | Size | Used | Available | Use% | Disk Type | Notes | |-------|--------|------|------|-----------|------|-----------|-------| | `/var/opt/gitlab` (data) | `/dev/nvme1n1` | **16TB** | **5.9TB** | 9.7TB | **38%** | Hyperdisk | Candidate for right-sizing | | `/var/log` (log) | `/dev/nvme2n1` | 246GB | 725MB | 245GB | 1% | Hyperdisk | **Out of scope — not to be changed in GPRD** | --- ### :dart: Proposed Areas for Optimization #### 1. :desktop_computer: Compute Right-Sizing (Primary Goal) The cluster was migrated to `c4-highmem-96` during the PG18 upgrade without a post-upgrade load-based right-sizing pass. The actual CPU and memory utilization needs to be reviewed against the current load profile and capacity planning projections. - Review CPU and memory utilization metrics for all 4 nodes post-PG18 upgrade - Evaluate whether a smaller `c4-highmem` or alternative machine type can support the production workload with appropriate headroom - **Backup node (`-102`) should be evaluated independently** — as a nofailover/noloadbalance node, it may be a candidate for a significantly smaller machine type - Factor in capacity planning projections and growth estimates before finalizing target machine type - Assess cost delta between current and proposed machine types #### 2. :floppy_disk: Data Disk Right-Sizing The data disk is a **16TB Hyperdisk at 38% utilization (~5.9TB used, ~9.7TB free)**. While this is not as extreme as the staging registry case, there is meaningful headroom to evaluate. - Determine appropriate target disk size based on actual data usage + projected growth buffer - Review provisioned IOPS and throughput settings for the Hyperdisk and right-size to match actual workload - Evaluate whether current Hyperdisk tier and provisioned parameters are appropriate for the production Sec workload - **Log disk is explicitly excluded** from this analysis --- ### :moneybag: Potential Cost Savings (GCP — us-east1) _To be updated once target disk size and VM type are determined. Use GCP Pricing Calculator to finalize numbers._ | Component | Current | Proposed | Monthly Savings | Annual Savings | |-----------|---------|----------|-----------------|----------------| | Compute — standard nodes (3 nodes × `c4-highmem-96`) | TBD | TBD | TBD | TBD | | Compute — backup node (1 node × `c4-highmem-96`) | TBD | TBD | TBD | TBD | | Data Disk (4 nodes × 16TB Hyperdisk) | TBD | TBD (right-sized) | TBD | TBD | | IOPS/Throughput optimization | TBD | TBD | TBD | TBD | | **TOTAL** | **TBD** | **TBD** | **TBD** | **TBD** | _Update this table with GCP cost calculations once target sizes are decided._ --- ### :clipboard: Action Items - [ ] **Analysis:** Review CPU, memory, and I/O utilization metrics for all 4 nodes post-PG18 upgrade - [ ] **Capacity planning:** Validate proposed changes against capacity planning projections and growth estimates - [ ] **Compute sizing:** Determine target machine type for standard nodes (Leader + standard Replicas) - [ ] **Backup node compute:** Independently evaluate and right-size the backup node (`-102`, nofailover/noloadbalance) - [ ] **Data disk sizing:** Determine target data disk size based on actual usage + growth buffer - [ ] **IOPS/Throughput:** Review and right-size provisioned Hyperdisk IOPS and throughput parameters - [ ] **GCP cost calculation:** Update the cost savings table above using GCP Pricing Calculator - [ ] **Change Request:** Create and execute CR(s) for the production cluster rebuild with right-sized compute and disks (reference CRs from [#510 (closed)](https://gitlab.com/gitlab-com/gl-infra/data-access/dbo/dbo-issue-tracker/-/work_items/510) and [#751](https://gitlab.com/gitlab-com/gl-infra/data-access/dbo/dbo-issue-tracker/-/work_items/751)) - [ ] **Validate:** Confirm cluster health, replication, and performance post-change - [ ] **Document:** Record actual savings realized and update the parent epic --- ### :link: References - [dbo-issue-tracker#510](https://gitlab.com/gitlab-com/gl-infra/data-access/dbo/dbo-issue-tracker/-/work_items/510) — Optimize Patroni Sec Cluster for Cost and Efficiency _(reference procedure and CRs)_ - [dbo-issue-tracker#751](https://gitlab.com/gitlab-com/gl-infra/data-access/dbo/dbo-issue-tracker/-/work_items/751) — Staging Registry Cluster Disk Right-Sizing and Compute Optimization _(reference procedure)_ - [Epic &61](https://gitlab.com/groups/gitlab-com/gl-infra/data-access/dbo/-/work_items/61) — Database Infrastructure Cost Optimization: Systematic Monitoring and Continuous Efficiency Improvements