Group alerts by service
Group alerts by the
type label using Alertmanager grouping to only send one alert per service.
Update template for slack notification
Update alert template for slack notification to specify which service is firing and the list of alerts that fired. https://prometheus.io/docs/alerting/latest/notifications/ is a good reference for alertmanager templating
Update silence button
Update the silence button to silence all the firing alerts for that
type instead of a single alert, for example
As seen in
gitlab-com/gl-infra&746 (closed) we
have multiple alerts paging the SRE on-call in a few seconds for the
same service. Instead of sending multiple pages which can be stressful,
distracting and unclear which one to take action on, send only 1 per
Check that alertmanager configuration was updated.
Check that the templates where updated correctly.
Check that the slack message is correct.
Check that the pagerduty message is correct.
Check that alerts in elastic search are correct 👉https://nonprod-log.gitlab.net/goto/2c6ee400-e8cc-11ec-b771-57a829f2c394