ProcessCommitWorker sending a large amount of Gitaly requests
Summary
We have ProcessCommitWorker
sending around 3.5k requests per job. In gitlab-com/gl-infra/production#15973 (comment 1454980744) we see a single requested triggering 5k ProcessCommitWorker
jobs, in total sending 180k Gitaly requests.
Single post_recive
:
Triggering 5k ProcessCommitWorker
Triggering 180k Gitaly requests in 5 minute
Previous infradev issues:
- https://gitlab.com/gitlab-org/gitlab/-/issues/407247+
- ProjectCacheWorker via `/api/v4/internal/post_r... (gitlab-foss#65381 - closed)
Impact
This results in saturation spikes on Gitaly because each ProcessCommitWorker
is sending so many requests to Gitaly in a single Gitaly repository.
Verification
- A single
post_recive
should trigger so many jobs -
ProcessCommitWorker
should have lower number ofgitaly_calls
👉 https://log.gprd.gitlab.net/goto/638b0770-1a6c-11ee-a017-0d32180b1390