Unified Backups: Investigate what is required to support object storage and cloud-native hybrid architectures

Context

To support Reference architecture: up to 2,000 users and Cloud Native Hybrid approach we need to introduce Object Storage support.

This still doesn't have any quirk around highly available Databases, so no proxy workaround required.

For architectures above 1K, we will focus on object storage on cloud vendor solutions such as S3 and GCS for files. This is to limit the scope while we build out the overall solution.

There is inconsistent support for backing up object storage amongst the cloud vendors. The costs for those that do offer solutions can vary significantly.

We want to avoid intermediate processing and storage as much as possible. For example, the current backup solution downloads the files in object storage to the toolbox pod, tars them and uploads them back to object storage. Avoiding such intermediate storage and processing can lead to significant time savings.

Proposal

Let's take the simplest and broadly supported approach to cover the broadest possible customer base - replicate the contents of the buckets to a backup bucket in another region.

The cloud vendors offer APIs for these services. The unified backup solution will integrate with these APIs to orchestrate copying of the data from the bucket(s) to the backup bucket. I believe all vendors support both full and incremental.

Since the volume of data can be expected to be large and take significant time to copy to the backup bucket(s), we must explore an incremental strategy for backing up object storage to save time by backing up on changes since the last backup and to save on storage space.

We will focus on GCS to start with as it serves an internal demand within GitLab as well as SM customers who have deployed their GitLab instance on Google Cloud.