DRI @rnienaber Git fetch traffic presents a scaling challenge for GitLab.com. We built the pack-objects cache &372 to improve the efficiency of generating Git fetch response data on Gitaly servers. In doing so we have discovered another bottleneck in the way we transfer Git fetch data with gRPC. In this epic we are exploring if we can do something about the gRPC bottleneck. In the experiments described in https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1046 we learned that even with a 100% pack-objects cache hit rate, Git fetch traffic on c2-standard-30 VM's (the type we use for Gitaly servers) stalls at 1.1 GB/s, with the server going into CPU saturation. **When we replace gRPC with plain TCP sockets, we reach 3.9 GB/s and the CPU utilization is around 60%**. Note that 4 GB/s is the network bandwidth limit imposed by GCP on the VM. ## Goals 1. Disable gitlab-org/gitlab [PRE_CLONE_SCRIPT](https://docs.gitlab.com/ee/development/pipelines.html#pre-clone-step) which currently acts as a band-aid for CI Git fetch performance limitations 1. Increase the speed of data transfer to match the GCP network bandwith limit ## Related incidents - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/4613 - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/4507 - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/4182 - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/4143 - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/4132 ## Change in Direction During the implementation of the original solution of this epic ("streamrpc") the Gitaly team raised concerns about the design of the solution. In the discussion that followed, Scalability and Gitaly have agreed on a new design based on [yamux](https://github.com/hashicorp/yamux), a network multiplexing library that Gitaly already uses for other purposes. Scalability has validated this approach by benchmarking prototype code that uses yamux. ## Getting to Production In order to get to production as quickly as possible, we have restructured the project to move some issues to post-deploy. These are listed in the `Follow up Item` below. ## Follow up Items The follow up items listed below are cleanup tasks that need to happen after the project has been shipped to Production. - https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1302: Make sure server returns Canceled gRPC code if Sidechannel client hangs up after sending a valid request body - https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1301 : How and when to deprecate PostUploadPack - https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1170 : Wrap up epic 463 (git fetch grpc bypass) - https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1364 : Remove PackObjectsHook Gitaly RPC - https://gitlab.com/gitlab-org/gitaly/-/issues/3003: Investigate if this project improved the situation described in this issue - https://gitlab.com/gitlab-org/gitaly/-/issues/3021: See if this project improved this situation ## Out of Scope - SSH traffic is out of scope. Our CI traffic uses HTTP and it is this traffic we are focused on. ## Summary 2021-11-09 The new Git HTTP fetch transport is on by default for all GitLab installations as of GitLab 14.5. We have been using it on GitLab.com since 2021-10-12. Thanks to the combination of the new transport and the pack-objects cache &372, we now no longer need the CI_PRE_CLONE_SCRIPT for gitlab-org/gitlab: that script has been disabled since 2021-10-13.