For GitLab.com, Gitaly currently allows enough `PostUploadPack` gRPC calls to run concurrently that they can collectively saturate the host's CPU or memory capacity. Each such gRPC call spawns a `git pack-objects` process, which tends to be CPU bound and can potentially run for multiple CPU-minutes and accumulate a significant amount of anonymous process-local memory. The problem tends to be rare because most git repos do not have a large complex graph of git objects. One example of a repo that does exhibit this behavior is the linux kernel source. Gitaly already supports concurrently limiting, so begin by reviewing that config for our production environment. We want to set the host-wide limit to not allow saturating all CPUs with concurrent `git pack-objects` processes. If we can also set a smaller concurrency limit on a per-namespace or per-project basis, that would allow one project to reach its limit without also starving other projects on the same Gitaly host. The problem can probably be reproduced by running `git clone` on a repo of the linux kernel source, but it may be necessary to do so using a stale checkout, such that many diffs need to be calculated. For background context, see: * Incident Review: https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/9550 * Incident: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/1774 For reference: This is a flame graph of host-wide CPU usage on a Gitaly node (`file-45`) during a timespan when numerous `git pack-objects` processes were collectively saturating the host's CPU capacity. As shown, the majority of on-CPU time was spent calculating the deltas to send as a response to the git client. Complete SVG of flame graph: [file-45.during_git_pack_objects_cpu_saturation.svg](/uploads/09e22726b6126fd05347b4c5b232492b/file-45.during_git_pack_objects_cpu_saturation.svg) Static image of flame graph: