Trim down list of labels in alert slack messages
Our alerts messages in slack are quite verbose as they include the full list of alert labels.
Example
Labels :label:
Alert_class: slo_violation
Alert_type: symptom
Component: workhorse
Env: gprd
Environment: gprd
Feature_category: not_owned
Monitor: global
Pager: pagerduty
Rules_domain: general
Severity: s2
Sli_type: apdex
Slo_alert: yes
Stage: cny
Tier: sv
Type: web
User_impacting: yes
Window: 6h
Many of these are interesting for long-term analysis, but not super useful for incident response.
Proposal
When responding to incidents, generally only a subset of labels is relevant. Here would be my proposed set of labels to include:
Component: workhorse
Env: gprd
Sli_type: apdex
Stage: cny
Tier: sv
Type: web
Window: 6h
We may want to consider removing the rest of the labels from these messages.
Results
-
Remove noisy labels in slack message 👉 gitlab-com/runbooks!4723 (merged) -
Remove firing
inside of pagerduty since it's repeating👉 gitlab-com/runbooks!4725 (merged)
Before | After |
---|---|
|
|
Edited by Steve Xuereb