fix: create dedicate alert for Alertmanager webhook

What

• Ignore webhook integration for AlertmanagerNotificationsFailing alert.
• Create a new alert AlertmanagerWebhookNotificationsFailing for webhook integration.

Why

webhook integration is only used for noncritical alerts such as #feed_alerts-general. This can get noisy and non-actionable since sometimes CloudFunctions can timeout, and it's too sensitive, resulting in non-actionable alerts for the on-call.

Looking at the last year this was triggered 31 times, and no action was taken.

• Logs: https://nonprod-log.gitlab.net/goto/b8d6c6a0-6591-11ed-9af2-6131f0ee4ce6
• Integrations that failed: https://thanos.gitlab.net/graph?g0.expr=%20%20%20%20%20%20sum%20by%20(integration)%20(%0A%20%20%20%20%20%20%20%20increase(alertmanager_notifications_failed_total%7Bintegration!%3D%22webhook%22%7D%5B10h%5D)%0A%20%20%20%20%20%20)%20%3E%204%0A&g0.tab=0&g0.stacked=0&g0.range_input=1y&g0.max_source_resolution=0s&g0.deduplicate=1&g0.partial_response=0&g0.store_matches=%5B%5D
  • Note: They don't even show up in the 1y data because they are so shorted lived.