Automatically disable failing webhoks
Currently when a webhook fails it will be retried a few times before ending up in the Dead queue within sidekiq. No matter how many times the hook fails it will always keep trying to send them, filling up the dead queue with lots of failed jobs the administrator has no real control over.
At present I will occasionally look for persistent offenders and remove them, but I think it would be better if GitLab tracks the number of times a webhook fails and disables them rather than removing them.
Track the number of times a webhook has failed and disable webhooks that are continuously failing.
Links / references
May need to implement #26146 as well to enable users to re-enable them.
What is it? Why should someone use this feature? What is the underlying (business) problem? How do you use this feature?
As an administrator of an active GitLab instance I am looking at the sidekiq queues for issues and at times the number of webhook failures masks other more important issues.
Make sure these are completed before closing the issue, with a link to the relevant commit.