Web Hooks - honor Retry-After header when handling 429 Too Many Requests

Proposal

We currently consider all 4xx responses for web-hooks to be evidence they are misconfigured. Too many in a row result in the hook being temporarily disabled.

However, the response 429 - Too many requests should get different behavior:

We should:

• Back-off immediately (not after 3 such responses), since we have been told to back-off
• Honour the Retry-After header

But we should still record the attempt as a failure, as the webhook was not delivered.

Implementation details

We would add some special handling of the hook failure state when the log_data shows that the response code was 429 and there was a Retry-After header:

• Add a new method to WebHook called #retry_after!(time). This method would:
  • Set the WebHook#disabled_until property to a time in future that honours Retry-After.
  • Set WebHook#recent_failures to WebHooks::AutoDisabling::TEMPORARILY_DISABLED_FAILURE_THRESHOLD so that the webhook is immediately backed off and will not retrigger until disabled_until has passed.
• Modify WebHooks::LogExecutionService#update_hook_failure_state to call #retry_after! instead of #backoff! when the response was 429 and there was a Retry-After header.

We would add documentation of how 429 lead to immediate disabling honouring the Retry-After header.