feat(ci-orchestration): alert on CI pipeline creation rate drops

Hordur Freyr Yngvasonrequested to merge
pipeline-creation-rate-baseline-alerts into request-rate-drop-runbook-and-absent-gating

The pipelines_created SLI in ci-orchestration has only a request rate, so a drop in pipeline creation never burns error budget. The traffic cessation alerts only fire when the total rate is exactly zero. Nothing covers a partial, per-source drop — INC-11596, where merge-request pipelines stopped while push pipelines kept flowing, no alert fired, and detection came from user reports.

Stacked on !12042 (merged), which adds the runbook field used here.

This MR opts pipelines_created into requestRateDropAlertConfig (!12011 (merged)):

requestRateDropAlertConfig: {
  by: 'source',
  selector: { source: { oneOf: ['push', 'merge_request_event', 'parent_pipeline'] } },
  threshold: 0.5,
  tenants: ['gitlab-gprd'],
  runbook: 'ci-orchestration/alerts/CiOrchestrationServicePipelinesCreatedRequestRateDrop/',
},

It generates CiOrchestrationServicePipelinesCreatedRequestRateDrop (s3, Slack, 15m), and adds a runbook for it. The runbook is linked from the alert only, not from the SLI's traffic cessation and absent alerts.

Decisions worth reviewing

  • Threshold 0.5 for all three sources. Replayed against gprd from 2026-06-04 to 2026-11-03, it caught all five pipeline-creation incidents in that window. The only other firings were seven for parent_pipeline, on Saturdays and US holidays. 0.35 removes those, at the cost of later detection in production#23194, where push dropped first. We start at 0.5 and tune from the Slack firings. Replay details are in the internal notes below.
  • Other sources (schedule, pipeline, api, trigger, …) are left out via selector: they are too bursty for a ratio threshold and stay covered at zero by …TrafficCessation.
  • Mimir data gaps. With !12042 (merged) an absent source only counts as zero while other sources are still reported. The six gaps in the replay window where every series disappeared (e.g. 2026-08-18, 2026-09-22, with up missing too) no longer fire this alert; TrafficAbsent covers them.
  • Slack-only for now. Promotion to paging once its firings show it is quiet outside real incidents.
  • Known gaps: a single cluster going quiet (~20% of volume) does not trip it. Detection latency is about 15–20 minutes. Multi-week holidays and daylight-saving changes push ratios down; the runbook covers silencing.

References

Edited by Hordur Freyr Yngvason

Merge request reports