Implement PostUploadPackWithSidechannel client in Workhorse

What does this MR do and why?

For gitlab-com/gl-infra/scalability#1220 (closed)

In gitlab-com/gl-infra&463 (closed), we are trying to address the bottleneck caused by gRPC in git fetch operations. A series of experiment can be found here. To tackle this bottleneck, we found a way to avoid gRPC when exchange "heavy" data. The idea is that when the Gitaly server is handling the incoming PostUploadPack call, it opens a "sidechannel" connection back to the client. This connection is supposed to be quick and effient.

• The implementation detailed of sidechannel inside Gitaly can be found at gitaly!3768 (merged)
• As the new approach uses a new protocol, completely incompatible with normal gRPC, we implemented a replacement for PostUploadPack RPC called PostUploadPackWithSidechannel. The implementation can be found at gitaly!3883 (merged)

This MR is implement the first client support for this sidechannel protocol. We introduce new feature flag workhorse_use_sidechannel. Workhorse fetches this feature flag via an auth API call to the upstream. If the flag is enabled, workhorse redirect uploadpack-related calls to use the new RPC.

Screenshots or screen recordings

This change affects workhorse only. There shouldn't be effect on the user side. I conducted a series of manual tests. I attached some screenshot to the logs as a result.

The logs produced by Workhorse and Gitaly when cloning a repository in the local environment with sidechannel:

Workhorse logs

Gitaly logs

Praefect logs

How to set up and validate locally

1. Check out this branch

2. This MR depends on gitaly!3900 (merged), which exports the interfaces to use in the clients. If that MR is not merged yet, we'll have to check out that branch in local Gitaly folder, and tweak the go module configuration of workhorse to point the package to local Gitaly folder. Otherwise, skip this step.

   go mod edit -replace gitlab.com/gitlab-org/gitaly/v14=/path/to/gitaly

3. Make sure that Praefect is disabled. Praefect support for sidechannel is pending at gitaly!3862 (merged)

4. Re-compile both Gitaly and Workrhorse binary.

5. Enable workhorse_use_sidechannel feature flag in the Rails console:

Feature.enable(:workhorse_use_sidechannel, Project.find_by_full_path("root/gitaly"))

1. Clone a random repository in the local environment via HTTP, for example:

git clone http://127.0.0.1:3000/root/gitaly.git

1. Everything should work as expected. Watch Workhorse and Gitaly logs to confirm they are using PostUploadPackWithSidechannel.
2. Clone another repository not having the flag enabled. Workhorse and Gitaly logs must indicate that they use original PostUploadPack RPC.

Benchmark in my local environment

The benchmark is cloning gitlab-org/gitlab repository multiple times. I didn't expect there is any performance gain in my local environment. It turns out the sidechannel RPC call is slight faster than the recent RPC. It's a good sign.

Terminal logs

Scenario	Test 1	Test 2	Test 3
PostUploadPack (without Praefect)	50 seconds	49 seconds	49 seconds
PostUploadPackWithSidechannel (without Praefect)	46 seconds	46 seconds	46 seconds
PostUploadPack (with Praefect)	52 seconds	51 seconds	52 seconds
PostUploadPackWithSidechannel (with Praefect)	47 seconds	47 seconds	48 seconds

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

• I have evaluated the MR acceptance checklist for this MR.