Kubernetes native CSI snapshots for Gitaly volumes for ops.gitlab.net
Previously in production-engineering#25057 we setup automated snapshots for `ops.gitlab.net`'s Gitaly volumes using [Backup for GKE](https://cloud.google.com/kubernetes-engine/docs/add-on/backup-for-gke/concepts/backup-for-gke).
The Gitaly team is now exploring options to backup and restore Gitaly volumes in https://gitlab.com/gitlab-org/gitaly/-/issues/6009, and has decided to go with [Kubernetes native CSI snapshots](https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/volume-snapshots) automated using [`snapscheduler`](https://backube.github.io/snapscheduler/). In the interest of testing this solution, dogfood it and save some costs, we can look at implementing it in parallel of, and eventually in place of, Backup for GKE for `ops.gitlab.net`.
**Scope:**
1. [ ] Deploy a `VolumeSnapshotClass` as part of our standard storage classes [here](https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-helmfiles/-/blob/master/releases/10-gitlab-storage/values.yaml)
2. [ ] Deploy [`snapscheduler`](https://backube.github.io/snapscheduler/) as an optional cluster extra ([already done in `ops-central`](https://gitlab.com/gitlab-org/gitaly/-/issues/6009#note_2227887000))
3. [ ] Deploy a `SnapshotSchedule` for Gitaly
4. [ ] Test snapshots and restores
6. [ ] Report results to the Gitaly team in https://gitlab.com/gitlab-org/gitaly/-/issues/6009
7. [ ] Decide whether to keep CSI snapshot or Backup for GKE, decommission the other one
8. [ ] Bonus point: automate volume restore tests, with metrics and alerts
epic