gitaly-hooks causes excessive CPU load
This issue is created to track the issue seen at one of our large customers.
Below are relevant comments extracted from a lengthy Slack thread:
We think this may be due to the hook used for the
pack-objects
cache. We have to marshal/unmarshal the pack file twice since the pack-objects hooks were added in 13.11.Here's the data flow as Matt Smiley explained it to me:
git pack-objects
writes its payload over a pipe to Gitaly- Gitaly gRPC encodes the pack to write to the
gitaly-hooks
process forked bygit upload-pack
gitaly-hooks
unmarshals the response and writes out togit upload-pack
, which writes to back to Gitaly again- Gitaly then marshals the pack and writes out its response to workhorse / shell
gitaly-hooks
is consuming more cycles that git itself in theperf
recording we have
26% of the time in
gitaly-hooks
is just spent spinning up the go runtime, these are ephemeral processes we fork rapidly (edited)
To summarize, though, it looks like, the go runtime overhead for running numerous short-lived
gitaly-hooks
processes appears to outweigh the benefit of the pack-objects cache. That overhead is being paid whether or not the cache is enabled.
- Old call chain:
grpc (PostUploadPack) -> gitaly -> git upload-pack -> git pack-objects
- New call chain:
grpc (PostUploadPack) -> gitaly -> git upload-pack -> gitaly-hooks git pack-objects -> gitaly (grpc PackObjectsHook) -> git pack-objects
The new call chain involves spawning a short-lived
gitaly-hooks
process for every incomingPostUploadPack
orSSHUploadPack
gRPC call.
In the corrected flamegraph, we can see that
gitaly-hooks
represents 35% of the non-idle CPU time on the host, and of that time, only 23% is spent in our application code. The other 77% ofgitaly-hooks
CPU time appears to be mostly spent in go runtime overhead.
interesting, 27% on
gitaly-hooks
and 22% on git
For reference, the 24% of samples spent in
swapper
are idle CPU time, so I subtracted that when I saidgitaly-hooks
used 35% of non-idle time (i.e. 35% of the 100-24=76)