Skip to content

Only record saturation rules in appropriate rulers

Bob Van Landuyt requested to merge bvl-cleanup-thanos-saturation-rules into master

Only record saturation rules in appropriate rulers

If a saturation point is evaluated in Prometheus, this is also the place where we evaluate the the alerts. So this is the place where the metadata needs to be recorded.

If a saturation point is evaluated in thanos, then we need to record the metadata there.

This also avoids adding recordings in reference architectures for services that don't exist these environments.

This is done by unifying the decision for recording into the saturationPoint and we'll only record if there are applicable services for the saturation point.

Thanos is an exception for all of the rule evaluations, because it overrides the static environment labels with a the thanos environment. The recording rule file for Thanos should not have any recordings or alerts for saturation points that don't apply to thanos.

Precursor to !5829 (merged), indirectly for https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/23722, but I think we want this regardless of the approach we take for code_suggestions.

Edited by Bob Van Landuyt

Merge request reports