Commit 22d6d316 authored by Craig Miskell's avatar Craig Miskell 💬

Fix the SidekiqJobsMetricsMissing alert to only alert when there *should* be...

Fix the SidekiqJobsMetricsMissing alert to only alert when there *should* be metrics there.  Fixes false positives on the prometheus-app instances which don't have the mtail job.

c.f. https://www.robustperception.io/absent-alerting-for-scraped-metrics for inspiration.  We have to ignore le, prog, and worker as those labels don't exist in the up metric.
parent a5116bf5
......@@ -10,7 +10,7 @@ groups:
runbook: troubleshooting/sidekiq_stats_no_longer_showing.md
title: Sidekiq stats failed to be scraped for the last minute
- alert: SidekiqJobsMetricsMissing
expr: absent(sidekiq_jobs_completion_time_seconds_bucket)
expr: up{job="mtail", type="sidekiq"} == 1 unless ignoring(le, prog, worker) sidekiq_jobs_completion_time_seconds_bucket
for: 2h
labels:
severity: s4
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment