Although not recommended or documented, it used to betechnically possible to deploy a gRPC-aware proxy between Gitaly andthe rest of GitLab. Examples of gRPC-aware proxies are NGINX andEnvoy. If self-managed instances currently use a gRPC-aware proxy forGitaly connections they should change their proxy configuration to useTCP or TLS proxying (OSI Layer 4) instead.Note that Gitaly Cluster has been incompatible with gRPC-awareproxies since GitLab 13.12. What changes is that _all_installations are now incompatible with gRPC-aware proxies, evenwithout Gitaly Cluster.The reasons for this change are outlined in [thisepic](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/463). Bysending some of our internal RPC traffic through a custom protocol weincreased throughput and reduced Go garbage collection latency.
Breaking Change
Yes. Users should remove the gRPC proxy between gitaly and the rest of GitLab, or reconfigure the proxy to work on the level of TCP/TLS sessions instead of gRPC requests.
Affected Topology
Self-managed
Affected Tier
All.
Checklist
@mention your stage's stable counterparts on this issue. For example, Customer Support, Customer Success (Technical Account Manager), Product Marketing Manager.
To see who the stable counterparts are for a product team visit product categories
If there is no stable counterpart listed for Sales/CS please mention @timtams
If there is no stable counterpart listed for Support please @mention@gitlab-com/support/managers
If there is no stable counterpart listed for Marketing please mention @cfoster3
@mention your GPM so that they are aware of planned deprecations. The goal is to have reviews happen at least two releases before the final removal of the feature or introduction of a breaking change.
Joshua Lambertmarked the checklist item @mention your GPM so that they are aware of planned deprecations. The goal is to have reviews happen at least two releases before the final removal of the feature or introduction of a breaking change. as completed
marked the checklist item @mention your GPM so that they are aware of planned deprecations. The goal is to have reviews happen at least two releases before the final removal of the feature or introduction of a breaking change. as completed
Joshua Lambertmarked the checklist item @mention your stage's stable counterparts on this issue. For example, Customer Support, Customer Success (Technical Account Manager), Product Marketing Manager. as completed
marked the checklist item @mention your stage's stable counterparts on this issue. For example, Customer Support, Customer Success (Technical Account Manager), Product Marketing Manager. as completed
@bmiller1 - perhaps @rnienaber or @jacobvosmaer-gitlab can confirm, but I believe we do not ship any proxies by default between gitaly and the rest of our services. So if the customer is following our reference architectures, they should be OK. If they have inserted a proxy, it would be good for them to see if it is specifically handling gRPC or if it can support binary proxying.
That is correct - a proxy is not the default setup between gitaly and everything else. The customer would have needed to set this up specifically. @jacobvosmaer-gitlab Do you have a simple way to describe a check for them?
I presume the context here is that customers want to upgrade from 14.x to 15.x and they want to make sure this deprecation does not affect them.
If you are on GitLab 14.4 or newer you can enable a feature flag that causes GitLab to use the new transport. This can be done on the level of individual projects so the risk of enabling this is low; there is no global impact.
Create a test project with some Git data in it (a README file is enough)
Verify that you can clone the repository with Git HTTP
Look for PostUploadPackWithSidechannel log messages in the Gitaly logs
If you see those log messages then that proves that Workhorse was able to use the new PostUploadPackWithSidechannel RPC. In turn, that proves that Workhorse was able to connect to Gitaly (or Praefect) using the new protocol and that means this deprecation is not affecting your GitLab installation.
A quicker test is to look for PostUploadPackWithSidechannel log messages in the Gitaly logs. If you see those then you are not using a gRPC proxy and then you don't have to worry about this deprecation.
@jacobvosmaer-gitlab - Yes, that is correct. Customers are preparing to upgrade to 15.x and, due to turnover, the are not always aware of configuration deviations made by former SRE/SA's.
@mbruemmer
I noticed this was moved to the backlog - but it isn't clear to me what that means. Are we backlogging comms to customers that this no longer works (since 13.12) or is there some actual code changes which nned to be made as part of this deprecation?
This had historically been added to 14.8, hadn't made the milestone and then GitLab Bot went on its way to update the milestone 30 times. In short: The milestone was just inaccurate and nobody looked at it for a few years.
We'll have to find out what's going on here first before assigning it again.