Investigate increase in prometheus disk space in zonal clusters

We received multiple pages for prometheus disk space for the zonal gprd clusters, which I made an incident for.

production#17181

For the short term, we increased disk space gitlab-com/gl-infra/k8s-workloads/gitlab-helmfiles!3737 (merged), but we need to figure out why there's an increase, and also, why tamland didn't complain.

We have k8s pvcs in tamland, but it looks like they're global per cluster, not per disk. https://gitlab-com.gitlab.io/gl-infra/tamland/kube.html#kube-service-kube_persistent_volume_claim_disk_space-resource-saturation

We should fix that too.