Only record saturation rules in appropriate rulers
Only record saturation rules in appropriate rulers
If a saturation point is evaluated in Prometheus, this is also the place where we evaluate the the alerts. So this is the place where the metadata needs to be recorded.
If a saturation point is evaluated in thanos, then we need to record the metadata there.
This also avoids adding recordings in reference architectures for services that don't exist these environments.
This is done by unifying the decision for recording into the
saturationPoint
and we'll only record if there are applicable
services for the saturation point.
Thanos is an exception for all of the rule evaluations, because it
overrides the static environment labels with a the thanos
environment. The recording rule file for Thanos should not have any
recordings or alerts for saturation points that don't apply to thanos.
Precursor to !5829 (merged), indirectly for https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/23722, but I think we want this regardless of the approach we take for code_suggestions.