Data recovery for Gitaly not possible on ops.gitlab.net

Gitaly is a SPOF and there is a single persistent disk for Gitaly, running in K8s for ops.gitlab.net in us-central1-a

source

We don't have any snapshot policies configured for this disk making it unlikely that we would be able to restore repos in the case of a zonal/regional outage or accidental deletion.

• Investigate options for persistent volume snapshots https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/volume-snapshots
• Investigate options for enabling deletion protection, if possible through GCP or Terraform config policy.

cc @ayeung @gsgl @devin who were involved in the migration work for ops.gitlab.net