webhook to send events to AWS SQS message queue

Description

Web hook is not a particularly reliable way to send events outside gitlab. One have to make sure the system which receive events is reliable enough (gitlab only retries a few times). This is not easy and adds unnecessarily costs.

When Gitlab is hosted on amazon it makes sense to send the events to amazon SQS, which offers reliable and very cheap way to get messages out of the system.

Proposal

Change webhooks system so that it not only supports http(s), but also sqs. Instead of putting an http URL, one would put a sqs queue URL e.g:

https+sqs://sqs.us-east-2.amazonaws.com/123456789012/MyQueue

or

sqs://sqs.us-east-2.amazonaws.com/123456789012/MyQueue

This also would open the future path to supporting other message queue protocol e.g

amqp://rabbit_mq_host/MyQueue

With this design, we can implement several webqueue protocol with limited code to duplicate. The UI remains the same, and will support system, groups and project hooks. Only the WebHookService needs to be changed in order to switch implementation according to URL protocol variation.

Links / references

I already have PoC patch for this, and I am waiting for legal approval to share it. it is a pretty straighforward use of require 'aws-sdk'