Guarantee post commit notification delivery to Rails in Gitaly
Gitaly calls into Rails through PostReceiveHook
to notify it of new writes. This notification is also used to trigger pipelines, and possibly other follow up tasks that Rails wants to do. The notification is not guaranteed to be delivered as it's attempted once and there are no further delivery attempts. This is not ideal as temporary failures in Gitaly or Rails may lead to the pipelines and other follow up tasks never running. Other impacted services include the Zoekt code search indexing which relies on notifications to trigger indexing. Missing a notification could lead to stale search indexes. There are likely more impacted tasks as well.
With &8911, we'll have each write persisted in the log. This opens up the possibility to retry the notification delivery from the log after recovery, giving us at-least-once delivery semantics. This would lead to a better reliability as tasks that are expected to run will eventually run.
As each write in the log has a unique log sequence number, we could use them to deduplicate notifications on the Rails side to get exactly-once delivery semantics. This would avoid running redundant work and make it easier for the downstream consumers to reason about the notifications.