ProcessCommitWorker sending a large amount of Gitaly requests

Summary

We have ProcessCommitWorker sending around 3.5k requests per job. In gitlab-com/gl-infra/production#15973 (comment 1454980744) we see a single requested triggering 5k ProcessCommitWorker jobs, in total sending 180k Gitaly requests.

Single post_recive:

Triggering 5k ProcessCommitWorker

source

Triggering 180k Gitaly requests in 5 minute

source

Previous infradev issues:

1. https://gitlab.com/gitlab-org/gitlab/-/issues/407247+
2. ProjectCacheWorker via `/api/v4/internal/post_r... (gitlab-foss#65381 - closed)

Impact

This results in saturation spikes on Gitaly because each ProcessCommitWorker is sending so many requests to Gitaly in a single Gitaly repository.

Verification

1. A single post_recive should trigger so many jobs
2. ProcessCommitWorker should have lower number of gitaly_calls 👉 https://log.gprd.gitlab.net/goto/638b0770-1a6c-11ee-a017-0d32180b1390