Specify web-hook retry policies
Problem to solve
Web-hooks have a few sharp edges, including failure modes. For large complex projects with 100's of developers, webhooks and custom integrations are important for many reasons, but that means that disabling hooks and failing hooks can cause loss of data and cause systems to break.
Users have no way to tell us how important their processes are, and we have no way to distinguish nice-to-have messages that will tolerate failure to essential messages that must be delivered eventually.
Proposal
Allow webhooks to be associated with retry policies.
When a webhook cannot be delivered, either due to failure or because the webhook is disabled, we can store the message, and enqueue it either for immediate retry or eventual delivery once the webhook is re-enabled, ensuring no dropped messages.
These retry policies could include:
- drop: as current - allow messages to be dropped.
- retry up to N times: attempt to send each message up to N times, then enqueue. Enqueue without retries if the webhook is disabled.
- enqueue: never retry, but place failed messages in a queue that a maintainer can manually set to send, or will be sent as soon as the webhook is re-enabed.
- something else?
Due to the need for this at larger organizations, and the cost of storing these records (and potential abuse), this would be a good candidate for an EE feature, especially if storage is allowed to be open-ended.