Assess latency impact on Code Suggestions when going through Gitlab.com
Problem
One of the biggest talking points when it comes to switching to go through the instance for Code Suggestions, is the added latency that this change in architecture will cause. There is a general understanding that increased latency will decrease acceptance rates of code suggestions. As a result, we need to first understand the added latency.
There were first explorations of the impact on Self-Managed when doing this switch. (see https://gitlab.com/gitlab-org/gitlab/-/issues/4182039)
Desired Outcome
We have p50 and p95 data on the latency of a code completion request for:
- Extension -> Model Gateway
- Extension -> Gitlab.com -> Model Gateway
Monitoring this will be a continued effort while both architectures are running, since changes to the architectures might change the numbers here as well.
Added information
Latency for Gateway can be found in https://log.gprd.gitlab.net/app/dashboards#/view/6c947f80-7c07-11ed-9f43-e3784d7fe3ca?_g=h@2294574
Next step
After gathering data on the latency, we'll need to figure out a way to roll this out to parts of our userbase to start getting data on the relationship between latency and the acceptance rate of suggestions.