Product webhooks are disabled and need manual intervention to be enabled
Webhooks on some projects are used to gather metrics on events like job failures. The webhook events are sent to the delivery-metrics server. However, if a few webhook event API calls receive error responses from the delivery-metrics server, the webhook gets disabled and needs manual intervention to be re-enabled.
When the webhook is disabled, we are not notified, so discovering that webhooks on a project have been disabled is also a problem.
The following docs provide some information about the automatic disabling of webhooks: https://docs.gitlab.com/ee/user/project/integrations/webhooks.html#failing-webhooks.
Project or group webhooks that fail four consecutive times are automatically disabled.
Project or group webhooks that return response codes in the 5xx range are understood to be failing intermittently and are temporarily disabled. These webhooks are initially disabled for one minute, which is extended on each subsequent failure up to a maximum of 24 hours.
Project or group webhooks that return response codes in the 4xx range are understood to be misconfigured and are permanently disabled until you manually re-enable them yourself.