[CI] Address Prometheus running out of disk space
Summary
Context: #3036 (comment 771275821)
Prometheus periodically runs out of disk space, and causes a CrashLoopBackOff which causes stable
releases to fail.
Steps to reproduce
See logs at https://gitlab.com/gitlab-org/charts/gitlab/-/jobs/2065407865
Configuration used
Current behavior
Prometheus periodically fails because it's out of disk space.
Expected behavior
Prometheus automatically cleans up data before running out of space.
Versions
- Platform:
- Cloud: GKE | EKS
Relevant logs
err="opening storage failed: open /data/wal/00000754: no space left on device"
Edited by Mitchell Nielsen