Online consistent backups
Online consistent backups
Release notes
Problem to solve
In order to execute a consistent backup today, we require to cut access to the environment, either via some form of downtime, or via enabling maintenance mode.
This is necessary to be able to get all the data in the same point in time.
As the datasets for backups become increasingly larger it becomes challenging to preserve data consistency in the backup as they may take several ours, making it impractical to schedule larger maintenance windows.
The longer the backup run takes the larger the RPO or the window of data loss.
We need to consider a separate solution for Unified Backups (local based) and Cloud Backups.
Proposal
Postgres already supports transaction logs. Gitaly will in the future support transaction logs. Each data type (Database, Repository, Files) backup can run in parallel and complete at different times. By including transaction logs in the backup it will be possible to compensate for the different completion times for each data type.
For PostgreSQL and Gitaly, we include transaction logs from the end of the respective backup until the consistency boundary (the longest-running backup).
In the example above we are assuming that object storage will take the longest time to backup given the volume of data. We are not proposing transaction logging for object storage in this proposal.
This is an alternative approach to quiescing the GitLab instance to take a consistent backup. It can be performed without having to pause operations and without impact on the developer experience.
For this feature, we will need access to the transaction logs for both PostgreSQL and Gitaly. For managed databases such as RDS and Cloud SQL this may not readily be available.
Intended users
Sidney (Systems Administrator)
Feature Usage Metrics
TBD
This page may contain information related to upcoming products, features and functionality. It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes. Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc.
- Show closed items