Streaming Git backups MVC
Problem to solve
Backing up a large GitLab instance before each upgrade is important, but the backup rake task (https://docs.gitlab.com/ee/raketasks/backup_restore.html#backup-strategy-option) doesn't scale to enormous instances. When I put my GitLab instance into maintenance mode, so that I can be guaranteed a completely consistent backup, I don't want to have to wait hours for the rake task to complete, or invent my own backup solution, that is probably incomplete and its own source of failure.
Further details
An experimental/prototype utility to demonstrate this was created !1244 (closed) and demoed https://www.youtube.com/watch?v=3eTJH7hPJqc
Proposal
When ever a write operation occurs, Gitaly should save an incremental snapshot of that write operation to object storage.
If a backup needs to be restored, the incremental snapshots can be replayed to create a complete copy.
Incremental snapshots will need to be collapsed at some frequency to prevent long chains of snapshots.