Webhook notifications with at-least-once delivery guarantee

Context

The container registry supports webhook notifications as described in https://docs.docker.com/registry/notifications/. This is currently used for delivering notifications to Rails whenever an image is pushed, but the information is only used to feed metrics (push count).

Related to gitlab#21042 (comment 542630004).

Problem

The as-is notification mechanism does not offer delivery guarantees, as notifications are staged in vulnerable per-instance in-memory queues. Because of this, it's risky to use these async notifications for any kind of data synchronization, as data inconsistencies may occur.

Solution

In order to provide the at-least-once delivery guarantee, we should use a reliable and persistent queueing system to store these messages and retry their delivery in case of failures or instance restarts.

This can be achieved by using either the metadata database or a separate store (e.g. Redis) to store and process messages from.

Helpful links

Edited by Tim Rizzi