Automatically disable failing webhoks
Description
Currently when a webhook fails it will be retried a few times before ending up in the Dead queue within sidekiq. No matter how many times the hook fails it will always keep trying to send them, filling up the dead queue with lots of failed jobs the administrator has no real control over.
At present I will occasionally look for persistent offenders and remove them, but I think it would be better if GitLab tracks the number of times a webhook fails and disables them rather than removing them.
Proposal
Track the number of times a webhook has failed and disable webhooks that are continuously failing.
Links / references
May need to implement #26146 (moved) as well to enable users to re-enable them.
Documentation blurb
Overview
What is it? Why should someone use this feature? What is the underlying (business) problem? How do you use this feature?
Use cases
As an administrator of an active GitLab instance I am looking at the sidekiq queues for issues and at times the number of webhook failures masks other more important issues.
Feature checklist
Make sure these are completed before closing the issue, with a link to the relevant commit.
-
Feature assurance -
Documentation -
Added to features.yml