Review Prometheus needed disk space (on Jan 2024)
Summary
Just running a sanity check on the required disk space for our stable Prometheus deployment, which currently is 8 Gi
.
Prometheus Disk Size calculation
Calculation Formula as per Prometheus 2.40 Storage docs:
needed_disk_space = retention_time_seconds * ingested_samples_per_second * bytes_per_sample
Values:
-
retention_time_seconds: 345600
(In our case it's set to 4 days. So4 * 24 * 60 * 60 seconds
) (--storage.tsdb.retention.time=4d
).
For the other 2 attributes I couldn't find official resources, but the community from Stack Exchange seems to be using the following:
-
ingested_samples_per_second
:rate(prometheus_tsdb_head_samples_appended_total[2h])
-
bytes_per_sample
:rate(prometheus_tsdb_compaction_chunk_size_bytes_sum[2h]) / rate(prometheus_tsdb_compaction_chunk_samples_sum[2h])
Alternatively, The Prometheus docs state that bytes_per_sample
usually averages between 1 to 2 bytes:
bytes_per_sample: 2
✅
Sanity check for cloud_native_v122 retention_time_seconds: 345600
ingested_samples_per_second: 8521.812464985995
bytes_per_sample: 0.9142178796945283
Calculation: 345600 * 8521 * 0.91 = 2.679.820.416 (~ 2.6 GB)
Alternatively with suggested bytes_per_sample: 2
: 5.889.715.200 (~ 5.9 GB)
⏲
Sanity check for cloud_native_v125 TBD. Our server were recently reset, so we don't have enough data.
✅
Sanity check for cloud_native_v126 retention_time_seconds: 345600
ingested_samples_per_second: 9617.657703081233
bytes_per_sample: 0.660892040678298
Calculation: 345600 * 9618 * 0,66 = 2.193.827.328 (~ 2.2 GB)
Alternatively with suggested bytes_per_sample: 2
: 6.647.961.600 (~ 6.7 GB)
Steps to reproduce
- Find the pod name of the prometheus server:
kubectl get pods -n helm-charts-win -l app=prometheus
. - Port forward:
kubectl --namespace helm-charts-win port-forward $POD_NAME 9090
.- Replace
$POD_NAME
with the value from 1.
- Replace
- Query the values to calculate the formula as explained above.