Send special payload when webhook becomes rate-limited, or disabled

About

Webhooks can be:

• rate-limited
• disabled temporarily
• disabled permanently

We currently display an alert message #331407 (closed) when a hook is in one of these states, however, two of these states are temporary:

• rate-limited
• disabled temporarily

It's very likely that a hook can be in one of those two states for a period of time, and blocked from triggering, and the webhook owner will not have seen the alert. They may not be aware that some hooks were "dropped".

This is particularly the case for rate-limited webhooks. It may be that a hook goes over its rate limit for a few minutes every day, and will be blocked from triggering during that period. We currently don't report that unless a user happens to visit the webhook edit page during the minute it is rate-limited for.

Proposal

We could send a one-off request to the hook URL containing a small payload that signals that hook is blocked, and why. For rate-limited hooks this request could include a "rate limit" header 429.

We could use a debouncing mechanism like Gitlab::ExclusiveLeave.throttle.

In some ways, this would be an alternative to the idea of notifying failures by email (mentioned in #331407 (closed)) - as we're not sure who to send an email to.

Users would handle the request however they wanted.

A problem is that a user can't necessarily do anything about it, but at least they will get notified each time it happens.