Some volumes get rapidly full when Thanos unit is enabled (using Minio)
Summary
The volumes used by Minio get rapidly full when Thanos unit is enabled. This was observed on a 3 nodes CAPO management cluster, after few days all volumes get nearly full.
It should be interesting to understand how they are filled and possible storage data quantity/capacity optimization to be done (volume size could be increased but we should care about the footpringt of such feature which already consumes 8 volumes of 3Gi each (24 Gi in total).
related references
Details
$ k get pvc -n minio-monitoring-tenant
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
data0-monitoring-pool-0-0 Bound pvc-066cdb87-68de-4761-bd78-0982ce6e4613 3Gi RWO cinder-csi 8d
data0-monitoring-pool-0-1 Bound pvc-20769cb8-1683-42ef-8444-073123f9ef3d 3Gi RWO cinder-csi 8d
data1-monitoring-pool-0-0 Bound pvc-b6e9770e-893d-4803-b0e9-850afb7d9dd9 3Gi RWO cinder-csi 8d
data1-monitoring-pool-0-1 Bound pvc-8263d819-1f03-4909-bbd0-139401d7accb 3Gi RWO cinder-csi 8d
data2-monitoring-pool-0-0 Bound pvc-ceb20824-912d-46cc-b36a-8814ad27201f 3Gi RWO cinder-csi 8d
data2-monitoring-pool-0-1 Bound pvc-6f29fcce-4f3f-44c1-90b5-c2eb3fef81a7 3Gi RWO cinder-csi 8d
data3-monitoring-pool-0-0 Bound pvc-86375362-a968-4ec5-8c6a-e7fdbf7f47e8 3Gi RWO cinder-csi 8d
data3-monitoring-pool-0-1 Bound pvc-e4160b3d-75f4-4daa-9200-1947e53494e1 3Gi RWO cinder-csi 8d| critical | KubePersistentVolumeFillingUp | The PersistentVolume claimed by data2-monitoring-pool-0-0 in Namespace minio-monitoring-tenant is only 0.1062% free. |
|---|---|---|
| critical | KubePersistentVolumeFillingUp | The PersistentVolume claimed by data3-monitoring-pool-0-1 in Namespace minio-monitoring-tenant is only 0.1033% free. |
| critical | KubePersistentVolumeFillingUp | The PersistentVolume claimed by data1-monitoring-pool-0-0 in Namespace minio-monitoring-tenant is only 0.1048% free. |
| critical | KubePersistentVolumeFillingUp | The PersistentVolume claimed by data3-monitoring-pool-0-0 in Namespace minio-monitoring-tenant is only 0.1038% free. |
| critical | KubePersistentVolumeFillingUp | The PersistentVolume claimed by data1-monitoring-pool-0-1 in Namespace minio-monitoring-tenant is only 0.1042% free. |
| critical | KubePersistentVolumeFillingUp | The PersistentVolume claimed by data0-monitoring-pool-0-1 in Namespace minio-monitoring-tenant is only 0.1041% free. |
| critical | KubePersistentVolumeFillingUp | The PersistentVolume claimed by data0-monitoring-pool-0-0 in Namespace minio-monitoring-tenant is only 0.1046% free. |
| critical | KubePersistentVolumeFillingUp | The PersistentVolume claimed by data2-monitoring-pool-0-1 in Namespace minio-monitoring-tenant is only 0.1042% free. |
Edited by Alin H