feat(ci-orchestration): alert on drops in successful pipeline completions

Hordur Freyr Yngvasonrequested to merge
pipelines-finished-rate-drop-alert into master

!11233 (merged) added a request rate drop alert for pipeline creation. This MR does the same for pipeline completion, on the pipelines_finished SLI:

requestRateDropAlertConfig: {
  by: 'status',
  selector: { status: 'success' },
  threshold: 0.5,
  tenants: ['gitlab-gprd'],
  runbook: 'ci-orchestration/alerts/CiOrchestrationServicePipelinesFinishedRequestRateDrop/',
},

It generates CiOrchestrationServicePipelinesFinishedRequestRateDrop (s3, Slack, 15m), and adds a runbook for it.

Baseline history

The replay used the raw pipelines_finished_total counter, which goes back to 2026-07-10. The alert's baseline comes from sli_aggregations:pipelines_finished_total:rate_5m, which only has data in gprd since 2026-11-02 19:05. The two match where they overlap (median ratio 1.00). Once this lands:

Period Baseline Behavior
Until about 2026-11-09 19:05 none Cannot fire
2026-11-09 to 2026-11-16 one week Fires, but one unusual week sets the bar
2026-11-16 to 2026-11-23 average of two weeks Mostly as replayed
From 2026-11-23 3-week median As replayed

US Thanksgiving (2026-11-26) falls right after the baseline is complete. Expect Slack firings from this and the creation drop alert that day, or plan a silence on both alertnames. !12058 (merged) corrects the framework docs, which said the baseline is complete as soon as the rules are deployed.

Decisions worth reviewing

  • Successes only, not the total. When pipelines fail in bulk, failed pipelines take the place of successful ones and the total hides the drop. In production#23018, failures were back to about 1.0× baseline from 23:30 while successes stayed at 0.07–0.13× for about 40 minutes. The total was at 0.2–0.3 over the same period.

  • by: 'status' with a selector, rather than no by. If the success series disappears while failed keeps reporting, it still counts as zero.

  • failed and canceled are left out. At 0.5, canceled fires dozens of times a week, and failed drops to 0.61 or lower on 5% of normal days.

  • Threshold 0.5. Replayed against gprd from 2026-07-31 to 2026-11-05 (97 days; the counter's history starts 2026-07-10):

    Incident Successes vs baseline (lowest) Fires at 0.5 Creation alert
    production#23018, 2026-09-24, HTTP 503 on gitlab.com 0.00 23:11 23:11
    production#23194, 2026-10-14, Redis 0.04 16:04 16:05
    production#23191, 2026-10-14, child pipelines not created 0.55 – fires on parent_pipeline
    production#23261, 2026-10-16, GitLab.com down 0.11, for about 10 minutes – –

    Outside incidents it fired only during the 2026-08-18 Sidekiq metrics gap. The lowest ratio on a normal day was 0.59, on US Labor Day. 0.65 would add Labor Day and a partial data gap on 2026-09-22.

  • Sidekiq metrics gaps fire this alert. The creation alert does not fire on them, because pipelines_created_total only comes from Sidekiq and the series disappears. pipelines_finished_total is also emitted by API and web, so in a Sidekiq-only gap the series stays present at about zero, and the !12042 (merged) absent check cannot tell it apart from a stall. That was one firing in 97 days. The runbook says to check for it first.

  • Known gaps: in every incident where completions dropped, creation dropped in the same minute, so the replay does not show this alert catching something the creation alert misses. The cases where it would add coverage, processing stalls and runners failing jobs in bulk, did not happen in the window. Failures limited to a few projects, like production#23257, do not move the global rate.

References

Edited by Hordur Freyr Yngvason

Merge request reports