Centralizing discussions around the Code Suggestions/Architecture proposals so that they are easier to consume and understand Proposals should: - Decrease latency - Apply to self managed **DRIs:** @jprovaznik for ~"group::code creation", @rzwambag for ~"group::cloud connector" #### Resources - [Why we are considering this only for Code Completion](https://gitlab.com/gitlab-org/gitlab/-/issues/434063#note_1688317891) - [Our current architecture and why we are considering a change](https://gitlab.com/gitlab-org/gitlab/-/issues/434063#note_1680810690) - [A conversation around a possible end goal for timings](https://gitlab.com/gitlab-org/gitlab/-/issues/434063#note_1682908608) - [Most recent latency investigation](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/2660) - [Multi-region AI Gateway deployments](https://gitlab.com/gitlab-org/gitlab/-/issues/434873) #### Latency breakdown by components Currently each code suggestion request (both completion and generation) goes through this chain of component: Client (+Language server) -> Gitlab(Rails+Workhorse) -> AI gateway -> AI provider ```mermaid flowchart LR C[Client/LS] --> |http keep-alive|W[Workhorse] subgraph sg1["515ms"] W --> |30ms|R[Rails 39ms] R --> |http, 75ms|A[AI Gateway 2ms] subgraph 370ms A --> |http|P[AI Provider] end end ``` Each of the component adds additional latency to the overall request. Latency differs for completion and generation requests (because different pre/post processing but mainly because different AI models): * ~150ms is spent on connection Client->Gitlab (this may differ and depends on client's connection, https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/2660#note_1681992931) * 515ms is request duration in workhorse, following contributes to this timing:m * 145ms is spent from reaching Workhorse server to reaching AI gateway - measured by substracting request duration on workhorse (515ms) and request duration on AI gateway (370ms). * 30ms is spent on sending request between Workhorse and Rails (https://gitlab.com/groups/gitlab-org/-/epics/12224#note_1770680849) * 39ms (measured by request duration on Rails server as it's then handed-over to Workhorse) is spent inside Rails app itself. So the remaining time (100ms) is spent on connection between Workhorse<->AI gateway. * the remaining time (75ms) is probably spent on Rails<->AI gateway connection (this is just assumption) * AI gateway adds minimum latency (difference between duration_s and inference_duration_s is 2ms) * 368ms (for code completion) or 4025ms (for code generation) is spent by serving request by AI provider Above timings are deduced from median times from .com logging (IOW when user uses SaaS), for self-managed this timing might be slightly different because of different "self-managed Gitlab <-> AI gateway" connection timing. Based on above, it means that user should typically receive completion response in ~665ms (or more depending on user's connection) and code creation response in ~4300ms. Though there is probably also some additional latency between client<->language server - getting some estimation for this is still TBD, but I expect this to be marginal. ### Proposed solution * code completion requests are sent directly from client to AI-gateway (skipping Rails) * other requests (code generation, any other requests for which we don't detect that are completions) are sent through Rails for additional pre-processing * this different handling will allow us to noticeably decrease latency (by 145ms) for code completions while still doing data enrichment for code generation (for which this additional latency is marginal given the overall time) * in combination with [multi-region AI gateway](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1206) we will be able to eliminate long-distance connection latency completely (shave off additional time from client connection) * for now we keep using HTTP protocol (with keep-alive enabled) - switching to websockets would not make much difference (assuming we use JWT for user authentication which should be cheap) * a downside of this approach is that a different request flow is used for code completions vs code generations (or chat requests) Simplified diagram (see https://gitlab.com/groups/gitlab-org/-/epics/13252 description for detailed sequence diagram): ```mermaid flowchart TD C[Client] -->|Get auth token| R[Rails] C -->|Completion request| A[AI Gateway] C -->|Generation request| R R -->|Generation request with additional enrichment| A A --> P[AI Providers] ``` ### Alternatives Other options are discussed in https://gitlab.com/groups/gitlab-org/-/epics/12224#note_1735794079 (and thread below) but there are major issues for each of this option.