Review Prometheus needed disk space (on Jan 2024)

Summary

Just running a sanity check on the required disk space for our stable Prometheus deployment, which currently is 8 Gi.

Prometheus Disk Size calculation

Calculation Formula as per Prometheus 2.40 Storage docs:

needed_disk_space = retention_time_seconds * ingested_samples_per_second * bytes_per_sample

Values:

• retention_time_seconds: 345600 (In our case it's set to 4 days. So 4 * 24 * 60 * 60 seconds) (--storage.tsdb.retention.time=4d).

For the other 2 attributes I couldn't find official resources, but the community from Stack Exchange seems to be using the following:

• ingested_samples_per_second: rate(prometheus_tsdb_head_samples_appended_total[2h])
• bytes_per_sample: rate(prometheus_tsdb_compaction_chunk_size_bytes_sum[2h]) / rate(prometheus_tsdb_compaction_chunk_samples_sum[2h])

Alternatively, The Prometheus docs state that bytes_per_sample usually averages between 1 to 2 bytes:

• bytes_per_sample: 2

Sanity check for cloud_native_v122 ✅

• retention_time_seconds: 345600
• ingested_samples_per_second: 8521.812464985995
• bytes_per_sample: 0.9142178796945283

Calculation: 345600 * 8521 * 0.91 = 2.679.820.416 (~ 2.6 GB)

Alternatively with suggested bytes_per_sample: 2: 5.889.715.200 (~ 5.9 GB)

Sanity check for cloud_native_v125 ⏲

TBD. Our server were recently reset, so we don't have enough data.

Sanity check for cloud_native_v126 ✅

• retention_time_seconds: 345600
• ingested_samples_per_second: 9617.657703081233
• bytes_per_sample: 0.660892040678298

Calculation: 345600 * 9618 * 0,66 = 2.193.827.328 (~ 2.2 GB)

Alternatively with suggested bytes_per_sample: 2: 6.647.961.600 (~ 6.7 GB)

Steps to reproduce

1. Find the pod name of the prometheus server: kubectl get pods -n helm-charts-win -l app=prometheus.
2. Port forward: kubectl --namespace helm-charts-win port-forward $POD_NAME 9090.
   • Replace $POD_NAME with the value from 1.
3. Query the values to calculate the formula as explained above.