## Context This meta-issue is to document experiments and collaborate on backup/restore for large PostgreSQL database in GitLab environments. ### Related pairing issues [pairing issues with :minidisc: reaction](https://gitlab.com/gitlab-com/support/support-pairing/-/work_items?sort=created_date&state=all&assignee_username[]=mlockhart&search=Anton&my_reaction_emoji=minidisc&first_page_size=20) ### Why This Matters We are missing a backup solution for Omnibus-bundled, self-managed GitLab database > 100GB. Our [current guidance](https://docs.gitlab.com/administration/backup_restore/backup_large_reference_architectures/#configure-backup-of-postgresql-data) recommends to use cloud-managed database services. However many self-managed customers are unwilling – or _unable_ – to use these, and even where they can, it requires migrating the database for situations where customers began with a smaller, bundled database that has since grown. We need robust and supportable backup/recovery procedures for large database deployments to: - Safely execute maintenance operations (OS upgrades, REINDEX, etc.) - Complete regular backups with a reasonable interruption window (minutes to hours, instead of hours to days), or _no interruption_ - Recover from failures without data loss - Support failover operations without compounding corruption - Nice to have: Enable point-in-time recovery for disaster scenarios One option is to use [Barman](https://pgbarman.org/) — "Backup and recovery manager". It's an open-source solution which is used in the industry for ["physical" backups](https://gitlab.com/gitlab-com/support/support-team-meta/-/work_items/7479#note_3196360127) of PostgreSQL databases, either single servers, or Patroni clusters. The "physical" backup approach using `pg_basebackup` and streaming replication, is the same as our current Geo approach, and faster than the "logical" backup of `pg_dump`. ### Alternatives - **Use GitLab Geo for backups**. Geo is explicitly NOT a backup solution. From the [Gitaly Geo capabilities documentation](https://docs.gitlab.com/ee/administration/gitaly/gitaly_geo_capabilities.html): "Geo is not intended to replace other backup/restore solutions. Because of replication lag and the possibility of replicating bad data from a primary, customers should also take regular backups of their primary site and test the restore process." - **Follow GitLab SaaS SREs and use [WAL-G](https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/patroni/postgresql-backups-wale-walg.md)**. This is a bespoke customization of WAL-G for `gitlab.com`, is less mature than barman, doesn't include ease-of-use features, and not intended for consumption by our customers - **Continue with the `pg_dump`-based [backup command](https://docs.gitlab.com/administration/backup_restore/backup_gitlab/)** for logical database backups, against our own advice for large systems - **Backup and restore by stopping the database and using filesystem facilities** like snapshots or `rsync`. It is fastest, but it requires a complete GitLab outage, or juggling Patroni replicas, to backup - **Let customers work this out for themselves**. We will still need to work with them during self-managed emergencies or incidents which require database recovery ## Outcome **Completed:** 2026-08-17 This investigation is complete. Engineering was escalated at director level but declined to pursue Barman integration into GitLab. Instead, the work was delivered through Support-owned resources: - **Support Knowledge Base Articles:** Merged in [gitlab-com/support/articles!420](https://gitlab.com/gitlab-com/support/articles/-/merge_requests/420) - **Training Module:** [PostgreSQL_backup_barman.md](https://gitlab.com/gitlab-com/support/support-training/-/blob/main/.gitlab/issue_templates/PostgreSQL_backup_barman.md) template for Support team training - **Support Escalation:** [#7639](https://gitlab.com/gitlab-com/support/support-team-meta/-/work_items/7639) tracks Support preparedness and performance testing results - **Organization-level Tracking:** [gitlab-operating-model#99](https://gitlab.com/groups/gitlab-operating-model/-/work_items/99) coordinates this effort across the organization The original documentation MR ([!229743](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/229743)) was closed in favor of the Knowledge Base approach. ## Investigation Summary ### Research Completed (Lab Book Issues [#260](https://gitlab.com/mlockhart/lab/-/work_items/260), [#262](https://gitlab.com/mlockhart/lab/-/work_items/262)) - [x] Barman architecture and capabilities assessment - [x] Comparison with existing backup solutions (Omnibus backups, WAL-G, pg_basebackup) - [x] Testing Barman installation and configuration in lab environment - [x] Validation of backup/recovery procedures - [x] Performance impact analysis - [x] Integration points with GitLab HA deployments ### Advantages of Barman - **Continuous WAL archiving** with point-in-time recovery (PITR) capability - **Parallel backup and recovery** operations for faster RTO/RPO - **Incremental backup support** reducing storage and bandwidth requirements - **Built-in backup validation** and integrity checks - **Centralized backup management** for multi-node PostgreSQL clusters - **Better suited for HA environments** than Omnibus backup tool - **Flexible scheduling** and retention policies ### Challenges & Considerations - Requires separate backup server/infrastructure - Additional operational complexity vs. Omnibus backups - Integration with existing GitLab deployment procedures - Training and documentation needs for support team - Cost/benefit analysis for different deployment sizes - Monitoring and alerting setup required ### Testing Completed - Lab environment setup with Barman - Backup and recovery procedures validated - Performance benchmarking (backup duration, recovery time) - Failure scenario testing - Integration testing with Patroni failover - Incremental backup validation ## Recommendations ### When to Use Barman - Large databases (> 100GB) - Large HA deployments, when not using a managed cloud database service - Customers requiring point-in-time recovery capability - Environments with strict RTO/RPO requirements - Multi-datacenter deployments with Geo replication ### When to Use Omnibus Backups - Small single-node or 2-node deployments - Customers with simple backup requirements - Environments with limited infrastructure - Development/staging environments ## Related Issues - **[Fieldnote #996](https://gitlab.com/gitlab-com/support/fieldnotes/-/issues/996):** [ZD 674293](https://gitlab.zendesk.com/agent/tickets/674293) / CI runner tags stop matching jobs after Patroni DB failover (CLOSED) - **[ZD 672154](https://gitlab.zendesk.com/agent/tickets/672154):** Backup system failures blocking remediation - **[ZD 675522](https://gitlab.zendesk.com/agent/tickets/675522):** Project-specific UI load failure after DB recovery ## Lab Book References - **[Lab Book Issue #260](https://gitlab.com/mlockhart/lab/-/work_items/260):** Initial Barman investigation and setup - **[Lab Book Issue #262](https://gitlab.com/mlockhart/lab/-/work_items/262):** Barman testing and validation procedures