Alert PatroniLongRunningTransactionDetected Not Paging On-Call

Summary

We have an alert PatroniLongRunningTransactionDetected that has the labels severity: "s2" and pager: "pagerduty" this means that it should be paging the on-call, but as we can see in the last fire it didn't:

source

Looking at the prod_pagerduty route it has the following matcher:

    matchers:
    - env=~"gprd|ops|thanos"
    - pager="pagerduty"

If we look at the slack screenshot we see that the env label is missing which is why it's not being routed correctly, we even see the env label missing in the alert:

source

Related Incident(s)

• production#18677 (closed)
  • Incident Review: production#18695 (closed)

Desired Outcome/Acceptance Criteria

Since this has severity: "s2" and pager: "pagerduty" it should page the on-call, rather then just leaving a message in slack.

Corrective Action Issue Checklist

• Link the incident(s) this corrective action arose from
• Give context for what problem this corrective action is trying to prevent re-occurring
• Assign a severity label (this is the highest sev of related incidents, defaults to 'severity::4')
• Assign a priority (this will default to 'Reliability::P4' but should match the severity of the related incident)
• Assign a service label
• Assign a team label