gitaly-hooks causes excessive CPU load

This issue is created to track the issue seen at one of our large customers.

Below are relevant comments extracted from a lengthy Slack thread:

We think this may be due to the hook used for the pack-objects cache. We have to marshal/unmarshal the pack file twice since the pack-objects hooks were added in 13.11.

Here's the data flow as Matt Smiley explained it to me:

• git pack-objects writes its payload over a pipe to Gitaly
• Gitaly gRPC encodes the pack to write to the gitaly-hooks process forked by git upload-pack
• gitaly-hooks unmarshals the response and writes out to git upload-pack, which writes to back to Gitaly again
• Gitaly then marshals the pack and writes out its response to workhorse / shell

gitaly-hooks is consuming more cycles that git itself in the perf recording we have

26% of the time in gitaly-hooks is just spent spinning up the go runtime, these are ephemeral processes we fork rapidly (edited)

To summarize, though, it looks like, the go runtime overhead for running numerous short-lived gitaly-hooks processes appears to outweigh the benefit of the pack-objects cache. That overhead is being paid whether or not the cache is enabled.

• Old call chain: grpc (PostUploadPack) -> gitaly -> git upload-pack -> git pack-objects
• New call chain: grpc (PostUploadPack) -> gitaly -> git upload-pack -> gitaly-hooks git pack-objects -> gitaly (grpc PackObjectsHook) -> git pack-objects

The new call chain involves spawning a short-lived gitaly-hooks process for every incoming PostUploadPack or SSHUploadPack gRPC call.

In the corrected flamegraph, we can see that gitaly-hooks represents 35% of the non-idle CPU time on the host, and of that time, only 23% is spent in our application code. The other 77% of gitaly-hooks CPU time appears to be mostly spent in go runtime overhead.

interesting, 27% on gitaly-hooks and 22% on git

For reference, the 24% of samples spent in swapper are idle CPU time, so I subtracted that when I said gitaly-hooks used 35% of non-idle time (i.e. 35% of the 100-24=76)