feat(slo-alerts): runbook field and absent gating for request rate drop alerts

Hordur Freyr Yngvasonrequested to merge
request-rate-drop-runbook-and-absent-gating into master

Two changes to requestRateDropAlertConfig (!12011 (merged)), from review on !11233 (merged).

Runbook per drop alert

alertRunbook is an SLI-level field, so setting it for the drop alert also redirects the SLI's TrafficCessation and TrafficAbsent alerts. The config now has its own runbook field:

requestRateDropAlertConfig: {
  ...
  runbook: 'ci-orchestration/alerts/CiOrchestrationServicePipelinesCreatedRequestRateDrop/',
},

When it is not set, the annotation falls back to alertRunbook, then to <type>/#alerts, as before.

Absent series only count as zero while the SLI is reported

Before, the alert counted an absent rate series as zero in every case. A Mimir ingestion gap then fired the alert for every value of by at once. A replay of pipelines_created showed six such gaps in five months.

Now the baseline * 0 branch only applies while the SLI's counter is still reported for another value of by. That is the lazily created series case this is meant for. The check uses the SLI rate without selector. When the whole SLI is absent, TrafficAbsent covers it.

(
  (
    gitlab_component_ops_anomaly:rate_5m{...}
    or
    (
      gitlab_component_ops_anomaly:rate_5m:baseline{...} * 0
      and on (env, environment)
      sum by (env,environment) (
        sli_aggregations:pipelines_created_total:rate_5m{env="gprd"}
      )
    )
  )
  /
  (gitlab_component_ops_anomaly:rate_5m:baseline{...} > 0)
) < 0.5

Without by there is no other value to check, so the or branch is not generated and an absent rate does not fire.

No generated rules change: no SLI on master sets requestRateDropAlertConfig yet. !11233 (merged) is stacked on this MR and picks it up.

Merge request reports