## Background Organized here: https://gitlab.com/gitlab-org/gitlab/-/issues/517990#background We reviewed several options, and decided to use Codestral to replace code-gecko v2, which will be discontinued on 2025-04-09. ## **Goals** Ensure Duo customers can continue using code completion beyond the code-gecko v2 discontinuation date without an interruption. * Customers should not have to take any explicit action (e.g. update GitLab version) to continue using code completion. * Customers can continue using Vertex AI, if that is their preferred subprocessor. * The customer's deployment option (e.g. .com, Dedicated, self-managed) should not impact their continued use of code completion. ## **Proposal** ### Model & Subprocessor * Use [Codestral](https://cloud.google.com/vertex-ai/generative-ai/docs/partner-models/mistral#codestral-2501) 25.01 hosted on Vertex AI. * This ensures we have a Vertex AI hosted option for customers who have not yet approved any alternate subprocessors. Codestral is currently [served from](https://cloud.google.com/vertex-ai/generative-ai/docs/partner-models/mistral#codestral-25.01) US and Europe, but not APAC: | **Region** | **Quota system** | **Supported context length** | |------------|------------------|------------------------------| | `us-central1` | 60 QPM, 400,000 TPM | 32,000 tokens | | `europe-west4` | 60 QPM, 400,000 TPM | 32,000 tokens | ### Request routing * Requests for code-gecko v2 are directed to Codestral, which occurs within the AI Gateway * Ensure the logic is documented in the handbook - and not only in code * Default code completion model per AIGW deployment/region * US and Europe: Codestral via Vertex * APAC: Qwen2.5 7B via Fireworks * Verify that when Ops override feature flag [`code_completion_model_opt_out_from_fireworks_qwen`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/176841) is enabled, then those requests should be served by Codestral and not Qwen * Self-managed customers should not have to update their GitLab version to ensure this routing logic is enforced <table> <tr> <th>AIGW Region</th> <th>Default model</th> <th>Exceptions</th> </tr> <tr> <td>US</td> <td>Codestral 25.01 via Vertex</td> <td></td> </tr> <tr> <td>Europe</td> <td>Codestral 25.01 via Vertex</td> <td></td> </tr> <tr> <td>APAC</td> <td>Qwen2.5 7B via Fireworks</td> <td> When `code_completion_model_opt_out_from_fireworks_qwen` is enabled, then requests are served by Codestral 25.01 via Vertex </td> </tr> </table> ### Evaluations **New evaluations** * Please run latency evaluations so we can document the expected latency profile. * Please run the quality evaluations so we can document the expected quality. **Prior data for reference** * We have prior [production data](https://gitlab.com/gitlab-org/gitlab/-/issues/480209#note_2236664137) from the initial Codestral version. * The acceptance rate was promising, and mostly consistent with code-gecko. * The latency slower than our targets. The 25.01 version is claimed to be 30-60% faster than the initial version. * We ran into [scaling issues when ramping up traffic](https://gitlab.com/gitlab-org/gitlab/-/issues/505509) - initially we thought this was due to saturated hardware (i.e. maxing out GPU capacity) but the root cause was a Vertex bug, which is now resolved. ### Request quotas * We need to request a Vertex quota increase to support our production volume. We have run into rate limiting based on the default quota. # Replace discontinued code-gecko v2 with Codestral - Rollout Plan <table> <tr> <th>Target Date</th> <th>Task</th> <th>Status</th> </tr> <tr> <td>2025-03-04</td> <td>Introduce feature flag and AIGW check</td> <td> :white_check_mark: </td> </tr> <tr> <td>2025-03-06</td> <td>Push feature flag to AIGW for .com users (Including indirect and direct access)</td> <td> :white_check_mark: </td> </tr> <tr> <td>2025-03-20</td> <td> Enable feature flag at 50% - enables change for .com customers Note: there was a subsequent rollback due to rate limiting issues </td> <td> :white_check_mark: </td> </tr> <tr> <td>2025-03-31</td> <td>Re-enable feature flag at 50%</td> <td> :white_check_mark: </td> </tr> <tr> <td>2025-04-01</td> <td>Enable feature flag at 100%</td> <td> :white_check_mark: </td> </tr> <tr> <td>2025-04-03</td> <td>Remove feature flag - enables change for self-managed and Dedicated customers</td> <td> :white_check_mark: </td> </tr> </table>