Kubernetes native CSI snapshots for Gitaly volumes for ops.gitlab.net

Previously in #25057 (closed) we setup automated snapshots for ops.gitlab.net's Gitaly volumes using Backup for GKE.

The Gitaly team is now exploring options to backup and restore Gitaly volumes in gitlab-org/gitaly#6009, and has decided to go with Kubernetes native CSI snapshots automated using snapscheduler. In the interest of testing this solution, dogfood it and save some costs, we can look at implementing it in parallel of, and eventually in place of, Backup for GKE for ops.gitlab.net.

Scope:

  1. Deploy a VolumeSnapshotClass as part of our standard storage classes here
  2. Deploy snapscheduler as an optional cluster extra (already done in ops-central)
  3. Deploy a SnapshotSchedule for Gitaly
  4. Test snapshots and restores
  5. Report results to the Gitaly team in gitlab-org/gitaly#6009
  6. Decide whether to keep CSI snapshot or Backup for GKE, decommission the other one
  7. Bonus point: automate volume restore tests, with metrics and alerts