Gitter Datadog alerts silenced in PagerDuty
DataDog has a multitude of "Monitors" configured which are supposed to send alerts to PagerDuty.
This integration is not working. Datadog triggers alerts but PagerDuty silences them by assigning them very low priority.
Example monitor: Email notifications - this monitor was failing and would point out our recent incident https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/10411
Proposed solution
Send all DataDog "Alert" level events to non-critical PagerDuty service (but one that will show up in our slack channel).
Every Datadog alert should be investigated and most likely deleted.
/cc @MadLittleMods