Create service slo observance rules in Grafana Mimir
In thanos-rules/service_slo_observance.yml
we add recording rules for whether or not a particular service is meeting SLO or not. This results in the slo_obersvation_status
recording rules per Service and SLO type (apdex/errors). The value is a 1 if the service meets SLO, and 0 if it doesn't. This is later used in an avg_over_time
for calculating the Service Level Availability.
In thanos, these recording rules record for all environments and services at once, when we migrate them to mimir, we should separate these by service.
These rules will be needed for our SLA calculation that is being migrated in #2770 (closed)