Skip to content

[CI] Shorten Prometheus Server retention to address storage issues

Mitchell Nielsen requested to merge adjust-prometheus-retention into master

What does this MR do?

Sets the Prometheus Server retention to 2 days, overriding the default of 15 days.

This effectively sets --storage.tsdb.retention.time.

Aims to adddress #3110 (closed) due to periodic failures where Prometheus would run out of storage.

In the docs I can see that --storage.tsdb.retention.size is also available, but is not configurable via the Helm chart. If it ever is, we could potentially match this value to the size of the PVC.

For reference: Prometheus Server Deployment.

(The size setting isn't configurable in main either as of 2022-02-08)

Related issues

Closes #3110 (closed)

Checklist

See Definition of done.

For anything in this list which will not be completed, please provide a reason in the MR discussion.

Required

  • Merge Request Title and Description are up to date, accurate, and descriptive
  • MR targeting the appropriate branch
  • MR has a green pipeline on GitLab.com

Expected (please provide an explanation if not completing)

  • Test plan indicating conditions for success has been posted and passes
  • Documentation created/updated
  • Tests added
  • Integration tests added to GitLab QA
  • Equivalent MR/issue for omnibus-gitlab opened
Edited by Mitchell Nielsen

Merge request reports