Create dead man's snitch alerts for Mimir
In https://gitlab.com/gitlab-com/runbooks/blob/a31d6977c3e0cf57b2cd0526f4f9b1272ca9ba3d/thanos-rules/metamon.yml#L1 as well as rules/default/prometheus-metamons.yml
we have alerts that report to deadmanssnitch.com, so we would be alerted when something no longer alerts.
We need to add something similar for Mimir, these rules should go in mimir-rules/metamonitoring
.
The runbook on these snitches can help define these: https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/monitoring/prometheus-snitch.md#setting-up-a-snitch
The alertmanager receiver is configured in alertmanger/alertmanager.jsonnet
, with an example configuration in alertmanager/dummy-secrets.jsonnet
. The actual secrets can be Vault. See alertmanager/readme.md
for more information.