gitaly-hooks causes excessive CPU load
This issue is created to track the issue seen at one of our large customers.
Below are relevant comments extracted from a lengthy Slack thread:
We think this may be due to the hook used for the
pack-objectscache. We have to marshal/unmarshal the pack file twice since the pack-objects hooks were added in 13.11.
Here's the data flow as Matt Smiley explained it to me:
git pack-objectswrites its payload over a pipe to Gitaly
- Gitaly gRPC encodes the pack to write to the
gitaly-hooksprocess forked by
gitaly-hooksunmarshals the response and writes out to
git upload-pack, which writes to back to Gitaly again
- Gitaly then marshals the pack and writes out its response to workhorse / shell
gitaly-hooksis consuming more cycles that git itself in the
perfrecording we have
26% of the time in
gitaly-hooksis just spent spinning up the go runtime, these are ephemeral processes we fork rapidly (edited)
To summarize, though, it looks like, the go runtime overhead for running numerous short-lived
gitaly-hooksprocesses appears to outweigh the benefit of the pack-objects cache. That overhead is being paid whether or not the cache is enabled.
- Old call chain:
grpc (PostUploadPack) -> gitaly -> git upload-pack -> git pack-objects
- New call chain:
grpc (PostUploadPack) -> gitaly -> git upload-pack -> gitaly-hooks git pack-objects -> gitaly (grpc PackObjectsHook) -> git pack-objects
The new call chain involves spawning a short-lived
gitaly-hooksprocess for every incoming
In the corrected flamegraph, we can see that
gitaly-hooksrepresents 35% of the non-idle CPU time on the host, and of that time, only 23% is spent in our application code. The other 77% of
gitaly-hooksCPU time appears to be mostly spent in go runtime overhead.
interesting, 27% on
gitaly-hooksand 22% on git
For reference, the 24% of samples spent in
swapperare idle CPU time, so I subtracted that when I said
gitaly-hooksused 35% of non-idle time (i.e. 35% of the 100-24=76)