Gitaly server side backups fail due to updates for LATEST file for GCS bucket that has an active retention policy
Release notes
Add support for immutable backups to remote storage that have retention policies for adding files but not modifying them.
Problem to solve
When using server-side backups for our Gitaly cluster and using remote storage to a GCS (Google Cloud Storage) bucket, we have a retention policy enabled on the bucket that allows adding of files, but not modifying those files. The reason we do this is so we have immutability for our backups which is a company policy to prevent tampering.
The problem is that for each manifest and hashed segment, there is a file called LATEST that gets updates every time we perform a backup. The repository backups seem to work fine, all except for the update of this file which fails the backup process as the file cannot be modified due to the retention period set on the bucket.
Proposal
Add a flag (e.g. -skip-update-latest, -skip-create-latest, -skip-create-pointer) that would not need to create or update the LATEST file (pointer) if using server-side backups and not doing incremental backups. Since when we do a restore, we specify the ID of the backup explicitly. There is a fallback described here. The error handling is found here.
Intended users
System Administrators