Allow longer timeouts for webhooks
Update - reasons to close this issue, from comment:
The timeout is configurable by an instance admin already.
For GitLab.com, the timeout duration is an important piece of our resilience to having webhooks perform well.
Our best-practice recommendation for webhooks is that they are acknowledged quickly by a receiver and then queued #356156 (comment 1915857458).
It is becoming clear that triage-ops will not be able to rapidly reduce response times (see: gitlab-org/quality/triage-ops#921 (comment 876808785)).
That project would greatly benefit from having support for longer response timeouts, and I suspect others would too. These projects would especially benefit from guarantees that these timeouts will not be reduced, which we may wish to do to reduce sidekiq queue lengths.
One option would be to allow webhooks to specify custom timeout durations (as we do for pipelines). Since this would have an impact on sidekiq, this should be discouraged, and one way to do so is to make this a paid feature, to reflect the impact on resources.
Proposal
Allow webhooks to specify custom timeout duration, ranging from 10 to 60,000 ms. (paid only)
Allow webhooks to assume success on timeout (fire-and-forget mode), which sets a short timeout (1sec?) and does not consider timeouts a failure, and does not store responses (this should be in FOSS, since we should actively encourage users to use this mode)