[prometheus] gprd release=prometheus-gitlab-app-1
What and Why
Issue: https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/24579
Needs:
!3427 (merged) merge and rebase this MR!3466 (merged)
Common changes and best practices
- Add memory limits. No having a memory limit in kubernetes increases the chance of being killed by the OOMKiller.
- Memory Request = Memory Limit - Best practice, again reduce impact of OOMKiller for these workloads.
- Enabled feature "memory-snapshot-on-shutdown" Requires Prometheus >= 2.30. Prometheus writes out a more raw snapshot of its current in-memory state upon shutdown, which can then be re-read into memory more efficiently when the server restarts. Reduce starts time in 50-80%. Faster restarts https://github.com/prometheus/prometheus/pull/7229
Summary of changes per environment and instance
Env | Cluster env | Prometheus instance | Memory requests | Memory limits | CPU requests | CPU Limits | Dashboard |
---|---|---|---|---|---|---|---|
gprd | gprd-gilab-gke | prometheus-prometheus-gitlab-app-1-pr-prometheus | 10Gi | 100Gi -> 10Gi | 0.8 -> 1 | 75 -> none | Link |
This MR should only affects to gprd (gprd-gitlab-gke) environment and only to (release=prometheus-gitlab-app-1).
About memory-snapshot-on-shutdown
- Faster restarts, quicker recovery. It also works when the different probes fails for different reasons and Prometheus is SIGTERM.
- Snapshots will take additional space.
- Depending on how many series you have and the write speed of the disk, shutdown can take a little time. Therefore, we need to adjust pod termination grace period but, that is a setting that still is not supported in the Prometheus Operator https://github.com/prometheus-operator/prometheus-operator/issues/3433. At the moment, is hardcode to 10m (600s) which I believe will be more than enough for this use case. As I rolling out the setting, will test this assumption.
Additional notes
CPU usage is less than half a core, but give prometheus less than a 1 cpu core is not a good idea.
Edited by Raúl Naveiras