## 1. Summary / Desired Outcome ### Problem Statement Across GitLab environments (gitlab.com, Dedicated, Cells/Cellular, Runway, future services), PostgreSQL usage is: * **Operationally fragmented** and inconsistently managed * Growing in **scale and cost** (\~$xxM/year infra footprint) * Exposed to increasing complexity as we add new products, deployment models, and providers Key challenge: > How do we scale database operations, reliability, and efficiency ? In this Epic, we want to build a unified Postgres service platform that standardizes and consolidates database operations across GitLab environments (SaaS, Dedicated, Self-managed), enabling scalable fleet management, consistent reliability, and efficient production operations. The platform will streamline provisioning, monitoring, alerting, and incident response, while reducing operational fragmentation and friction across teams. It enables GitLab to manage a significantly larger PostgreSQL fleet with improved reliability, stronger standardization, and better long-term cost efficiency. This means: * Scaling from ad‑hoc / bespoke database setups → **hundreds to thousands of standardized clusters** * Reducing operational complexity and manual intervention * Establishing a predictable, testable, and automated **database lifecycle** * Creating a foundation for **cost optimization** and future platform capabilities ### ### **Success Metrics (quantifiable only)** 1. **Provisioning Time** * P50 / P95 database provisioning time ↓ (target: \< 15 min P50) 2. **Upgrade Success Rate** * % of automated upgrades completed without manual intervention * Target: \> 99% 3. **Incident Rate** * of Sev2+ incidents per cluster * Target: ↓ 50% 4. **Recovery Time (MTTR)** * Mean time to recover from cluster-level failure * Target: \< 30 min 5. **Operational Coverage** * % of clusters managed via standardized platform * Target: \> 50% ### Guiding Principles * **Outcome‑driven, not tool‑driven**\ Define capabilities and outcomes first; choose technologies that meet them. * **Fleet‑first**\ Optimize for managing _many_ clusters, not one perfect snowflake. * **Failure‑mode driven**\ Evaluate designs by their behavior under failure (upgrades, partitions, control‑plane issues), not feature checklists. * **Phased delivery**\ Ship a working MVP quickly, then iterate. * **Vendor‑agnostic (initially)**\ Avoid premature lock‑in to specific vendors or products; keep credible exit paths. * **Shared patterns over bespoke solutions**\ Maximize reusable architecture, infra, runbooks, standards, and tools. ## **2. Requirements** ### **Must Have** * DB HA Cluster-based Postgres deployment (not single instance) * Automated provisioning + lifecycle management (create / scale / delete) * HA with automatic failover (no manual intervention required) * Integrated monitoring + alerting (standardized signals) * Backup + restore (PITR) * Upgrade orchestration (major + minor, low/no downtime) * First‑class **connection pooling (Zero‑downtime / near zero‑downtime cutover** support (drain, pause/resume) * Multi-region / DR topology * Multi Cloud --- ### **Nice to Have** * autohealing system * Cost-aware placement / auto rebalancing * Performance auto-tuning (storage / memory) * Workload-aware scaling * Integration with AI-driven ops / anomaly detection --- ### **Not Required** * Multi-writer / active-active * Full replacement of existing GitLab.com DB stack * Custom storage engine / deep DB kernel changes/customization ## **3. Execution Plan (3 Quarters)** --- ## **Q1: Discovery + Convergence** ### **Phase A – External Provider Investigation** Evaluate: * Managed DBaaS services (e.g., Cloud SQL for Postgres, AlloyDB Omni/GDC, Yugabyte Cloud? \<compatibility\> , RDS? ) * PGaaS offering (Severalnines, PlanetScale Postgres, Tessell, etc.) * Open-source platforms (CNPG, Crunchy, YugabyteDB etc.) * Internal existing solutions? --- ### **Phase B – Internal User & Workload, Pattern Discovery (In parallel with Phase A)** ### **Interview / Survey Questions** ### **1. Current Usage** * What database are you using today? * Deployment model (VM / k8s / managed)? * Scale (DB size, QPS, connections)? ### **2. Pain Points** * What is the most painful operational task today? * Where do incidents typically come from? * What breaks most often? ### **3. Dependencies** * What systems/tools are tightly coupled? (monitoring, CI, backup) * Any hard compliance / security constraints? ### **4. Required Features (Must-have validation)** * HA / failover * Backup / restore * Upgrade automation * Observability * Scaling * Multi-region :point_right: Can ask question like ： > “If this is missing, you cannot run your service?” ### **5. SLO / Expectations** * Acceptable downtime? * Acceptable data loss? * Provisioning expectation? --- ## **Q2: Solution Convergence + Validation** ### Step 1 – Narrow to **3 candidate architectures based on invesitgation** Criteria: * Coverage of Must Have * Feasibility * Operational complexity --- ### Step 2 – Failure Mode Testing (core measure) Define **6–7 mandatory test cases** and test the 3 solutions: 1. Node failure (primary crash) 2. AZ failure 3. Network partition (split brain scenario) 4. Backup + restore (full + point-in-time) 5. Upgrade (major version) 6. Scale up/down (storage / compute) 7. Monitoring / alert correctness :point_right: Each test case must pass to be qualified as a candidate --- ### Step 3 – Selection Select **Primary Option** based on: * Max Must-Have coverage * Pass rate of test cases * Simplicity / operability --- ## **Q3: MVP Build + Gap Closure** ### Step 1 – Gap Analysis For selected solution: * Which Must Have are NOT met? * Can we: * Configure? * Extend? * Need to build? --- ### Step 2 – MVP Implementation Deliver: * Provisioning workflow * Upgrade workflow * HA + backup baseline * Monitoring integration --- ### Step 3 – Early Adoption * Target: **new workloads first** * Gradual expansion to: * Dedicated * Existing systems --- # **High-Level Plan in a nutshell (3 Quarters)** <div> ```mermaid gantt title PG Services Platform Execution Plan - 3 Quarters dateFormat YYYY-MM-DD axisFormat %b %Y section Q1 - Discovery + Convergence External provider research :q1a, 2026-05-01, 8w Internal user/workload discovery :q1b, 2026-05-01, 8w Must-have requirement alignment :q1c, after q1a, 4w Candidate capability matrix :q1d, after q1b, 4w section Q2 - Architecture Validation Narrow to 3 candidate architectures :q2a, 2026-08-01, 4w Design failure-mode test cases :q2b, after q2a, 3w Execute failure-mode testing :q2c, after q2b, 6w Score candidates and select option :q2d, after q2c, 3w section Q3 - MVP + Gap Closure Gap analysis for selected option :q3a, 2026-11-01, 3w Build MVP platform capabilities :q3b, after q3a, 8w Implement missing must-have gaps :q3c, after q3a, 8w Early adopter rollout :q3d, after q3b, 4w MVP review and next-phase planning :q3e, after q3d, 2w ``` </div>