## Problem Currently, all requests to `code-suggestions/completions` are automatically routed to Google's `code-gecko` model regardless of the surrounding context of the code. `code-gecko` is fast (for an LLM) but has only very limited input and output tokens (2048/64) compared to e.g. `code-bison` (6144/2048) or `text-bison`(8192/1024). In practice this means we are limited on the context we can send to the LLM, but the even more drastic difference is in the output that the LLM can provide. `code-bison` can write 32 times the amount of code based on a single request compared to `code-gecko`. In practice, this means that `code-gecko` is well suited to e.g. complete a line or maybe a short function, but realistically cannot generate for example a whole file based on a natural language input. ## Desired Outcome We automatically differentiate between code generation and code completion based on the context of the suggestion request (code before/after cursor) and send it to the appropriate model accordingly. ## Suggested Solution ### How to differentiate between Code-Completion and Code-Generation The differentiation between completion and generation should happen in iterative steps: 1. Identify based on a secret word, for example: ```javascript // Gitlab Duo: Function to determine if a file is empty ``` 2. Identify if there are only comments in the file similar to https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist/-/issues/221 3. Identify if there are no complete functions, like suggested in https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist/-/issues/202 4. Further iterations ### Where to differentiate between Code-Completion and Code-Generation In theory we have three different places where we could differentiate between the two: Extension, Gitlab Instance, AI-Gateway. We suggest, to do the differentation either in the Extension or the Gitlab Instance for three main reasons: 1. The limited available people familiar with the Gateway can continue working on Code-Completion prompt improvements separately without being affected by these changes. 1. The limited available people familiar with the extensions can continue working on unifying features into the language server. 2. Having a separate endpoint available for code-generation from within the instance will allow us to also use code generation in other contexts (e.g. as a tool for the Gitlab Duo Chat) 3. If we already know in the instance whether this is a request for Code Generation it makes more sense to immediately add more information from the repository. If we don't know it there we otherwise might waste response time providing information that is not going to be used in the downstream prompt. **Important:** Moving prompting for code generation into the instance is a two-way door decision. We believe that this lets us iterate more quickly and in parallel while getting an understanding whether working with Treesitter in Ruby is feasible as well as understanding how a Code Generation LLM behaves. The prompt can always be migrated to the extension or the gateway at a later point. We decided on the following setup for now #### Differentation being done in the Instance ```mermaid graph TD A[Extension]--->B subgraph Instance B[code-suggestions/completions] end subgraph AI Gateway B---> D[v2/completions] B---> E[code-bison proxy ] end ``` ##### Which model to use Because of the latest developments on `code-bison` and further expected improvements we suggest to use `code-bison`