<!-- triage-serverless v3 PLEASE DO NOT REMOVE THIS SECTION --> *This page may contain information related to upcoming products, features and functionality. It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes. Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc.* <!-- triage-serverless v3 PLEASE DO NOT REMOVE THIS SECTION --> Possible future iteration: [Object-level incremental backups](https://gitlab.com/groups/gitlab-org/-/epics/3586) ## Problem to solve When I am operating a GitLab instance with large amounts of Git data, the backup rake task is not usable. I need to have reliable up to date backups that I perform at least as often as I upgrade GitLab. Taking a complete snapshot of a large instance is **slow** (12GB took 15 mins) - it can take days to bundle and compress each repo one by one. This is very impractical because normally you want regular backups (e.g. daily or weekly), and it could take much longer than that actually take the backup. This is a disaster recovery method that allows you to back up repositories stored in Gitaly Cluster. It will be an alternative to the commonly used disaster recovery disc snapshot method. For context, Gitaly Cluster is a highly available option for git repository storage. Because of its distributed nature, Gitaly Cluster is challenging to back up using disc snapshots because during the snapshot, the discs will continue to change. To utilize disc snapshots, downtime is required. Incremental backups aim to provide a reliable disaster recovery method with zero downtime, lower operating costs, and simpler operability for Gitaly Cluster. ## Further details Creating the backup, particularly with regards to the repositories causes **heavy load** on various components of GitLab creating bundles, and moving data around. High resource utilization caused by backups can significantly impact application performance. When backups become so large and take so long, they will be **inconsistent** and very difficult to verify the integrity of. I need be able to create a consistent backup without downtime. Current workarounds: - GitLab Geo: use secondary as a backup - Geo is not a backup strategy. A secondary provides a singe point in time for recovery from the point replication is turned off. For upgrades this might be okay with careful coordination. - Does not provide multiple points in time of recovery - Expensive to run and operate - Disk snapshots - Public clouds support disk snapshots, but these are not guaranteed to be consistent - Depends on infrastructure layer, and custom scripts/process for backup/recovery - rsync - Not guaranteed to be consistent - https://docs.gitlab.com/ee/administration/operations/moving_repositories.html - https://gitlab.com/gitlab-org/gitlab/-/blob/master/bin/parallel-rsync-repos - dedicated Gitaly replica for backups - use load balancers to direct real traffic to primary nodes, and then use NFS and replica nodes for creating the backup. This prevents extreme loads degrading application performance ## Direction Large customers should have a simple, fast, reliable, and documented method for taking consistent backups. > :point_up: **Note:** improving the performance of the restore process isn't within scope of this epic. Restoring extremely large amounts of data into an empty environment will take some time. When improving repository backup performance, we will be mindful not to design a system that will be slow to restore. ## Problem Validation Problem validation is underway in [it's own issue](https://gitlab.com/gitlab-org/gitaly/-/issues/3885) ## Further details On GitLab.com ~"internal customer" we have hundreds of TB of repositories that we need to backup, and recover from. At the moment we use disk snapshots taken every 4 hours. This is not a perfect solution because the backup can be inconsistent as the disk may continue to change while the snapshot is being taken. The current proposed solution is to use ZFS and ARC, but this is complicated and hard for our customers to use. See https://about.gitlab.com/handbook/engineering/infrastructure/library/storage/block/repositories/zfs-repo-storage/ Other customers ~customer need to roll their own backup strategies for Git data because GitLab does not provide a solution out of the box. **We should be building a backup solution that works for large customers and GitLab.com** ### Analysis GitLab.com currently hosts about 500TB of Git repositories. These backed up in full every 24 hours, and incremental backups are taken every 4 hours using disk snapshots. Over the last 7 days, the repositories that have been updated total about 0.7% of the total volume. This means that taking snapshots of only the repositories that have changed since the last full backup should be very small. | Period | % size repositories changed | % number repositories change | |---|---|---| | 7 days | 0.7% | 0.9% | | 24 hours | 0.06% | 0.09% | | 12 hours | 0.02% | 0.01% | ## Proposal Build an incremental backup solution that is/allows: - **consistent** - `git bundle` produces consistent complete backups, and incremental backups - **fast** - so they can be taken frequently. The direction of this feature should support a continuous backup strategy, like streaming to object storage. - **cost/resource efficient** - frequently or continuously backing up the Git repository data shouldn't require provisioning more Gitaly nodes, or a Geo secondary. - **verifiable** - there should be features that support a posture of verification, so that backups aren't found to be bad when they are needed. - **point in time recovery** - it should be possible to recover to multiple points in time The backup solution should not: - require object storage, although there will likely be some object storage specific optimizations - custom backup formats For compatibility and consistency, we should build on top of `git bundle`. A minimal solution would need: - incrementally write backups to an external location, probably object storage - administrative command/tools for verifying the backups are up to date - tools or process for verifying the backups are correct (not corrupt) - tools for restoring a repository from backup ## Customers - https://gitlab.lightning.force.com/lightning/r/Account/00161000002wZ7ZAAU/view ~"customer+" - https://gitlab.lightning.force.com/lightning/r/Account/00161000004akIxAAI/view ~"customer+" - https://gitlab.lightning.force.com/lightning/r/Account/00161000004zrG3AAI/view ~"customer" - https://gitlab.lightning.force.com/lightning/r/Account/00161000005cxCtAAI/view ~customer - https://gitlab.lightning.force.com/lightning/r/Account/0016100001F4x1EAAR/view ~customer - https://gitlab.lightning.force.com/lightning/r/Account/0016100000ZxrqHAAR/view ~customer https://gitlab.zendesk.com/agent/tickets/121701 - https://gitlab.lightning.force.com/lightning/r/Account/0016100001I755jAAB/view ~customer ## Links / references