<!--The format of the title should be Title should follow the format `Breaking change exception: Description`--> # Breaking Change Exception Request ## Executive summary As part of the larger cleanup of [consolidating API endpoints for Duo Code Suggestions](https://gitlab.com/gitlab-org/gitlab/-/work_items/547122) to version 4 of the API, we will be removing support for v2 and v3 callers in the AI Gateway. This change is part of a broader migration already underway across GitLab Rails, AI Gateway, `gitlab-lsp`, and IDE integrations, with GitLab.com being updated first and the eventual cleanup explicitly gated on the AI Gateway two-major-version compatibility policy and observed traffic. The breaking event is not the migration itself, but the eventual removal of the legacy endpoints for any remaining callers that still depend on them directly, more specifically for self managed and dedicated customers. ### Does your breaking change meet one of these three criteria? 1. The impact of the breaking change has been **fully mitigated via an automated migration** that requires no action from the customer. Not fully. We can automatically migrate GitLab.com managed traffic to v4, but we cannot automatically migrate self-managed customers or any third-party or older IDE clients that still call legacy endpoints directly. 2. The breaking change will have **negligible customer impact** as measured by actual product usage tracking across Self-Managed, GitLab.com, and Dedicated. For instance if it impacts less than 1% of the GitLab customer base. Yes. The goal is to have negligible customer impact by the time the endpoints are removed. Gitlab.com will be automatically migrated, and the two version deprecation policy will provide enough time for legacy clients to upgrade. Without upgrading, the only clients that will be affected will be self managed and dedicated customers that use the Gitlab provisioned AI gateway but are on an older version of Gitlab (19.0 or below). 3. The breaking change is being implemented due to a **significant security risk**- Severity 1 or 2. No. ## Impact assessment ### How many users and customers are impacted? What Tier/ARR of customer? Code Suggestions is available on GitLab.com, GitLab Self-Managed, and GitLab Dedicated. The [code suggestions telemetry dashboard](https://log.gprd.gitlab.net/app/dashboards#/view/6c947f80-7c07-11ed-9f43-e3784d7fe3ca?_g=%28refreshInterval:%28pause:!t,value:0%29,time:%28from:now-6h,to:now%29%29) shows the usage and provides indicators of how many v2, v3 and v4 callers are present. ### Can we get the same outcome without a breaking change? Not exactly, since this is a breaking change due to a deprecation of legacy endpoints that will be eventually removed. ### When do you want to make the breaking change? Target removal milestone: GitLab 21.0. Proposed sequence: * Public deprecation announcement in GitLab 19.1. * Continued migration and monitoring during 19.x and 20.x. * Removal of legacy v2/v3 endpoints in GitLab 21.0, once the policy window and traffic criteria are satisfied. ### What is the alternative for customers to do the same job the change will break? There is no alternative planned, but any self managed and dedicated customers that upgrade within the two version window will experience no issues. ### How difficult is it for customers to migrate to the alternative? Is there a migration plan? N/A ### Does this breaking change interact with [other proposed or approved breaking changes?](https://gitlab.com/gitlab-com/Product/-/boards/9741901) No. ## Rollout & Communication plan 1. Migrate GitLab-controlled traffic to v4 first, with no noticeable changes for customers. 2. Announce the deprecation of the v2 and v3 AI Gateway endpoints 3. Monitor the code suggestions telemetry dashboard until remaining legacy traffic is negligible, and use that data to drive targeted customer communication and approval discussions. 4. Do not remove legacy routes before the two-major-version compatibility window and traffic thresholds are met. ### Have we handled similar breaking changes in the past? What can we learn from the success or failure of those methods? Any similar breaking change due to a legacy deprecation can be used as a reference, and the two version window (approximately 2 years) is enough time to allow all customers to upgrade. ### Are you employing any strategies to lessen the customer or support impact? * We are migrating GitLab.com first, reducing risk on GitLab-managed traffic. * We are explicitly using observability to measure remaining v2/v3 usage before cleanup. * We are not proposing immediate removal; instead we plan deprecation first and removal only after the compatibility window and traffic thresholds are met. ### Internal Communication Please reference the [Internal Communication](https://internal.gitlab.com/handbook/engineering/r-and-d-pmo/knowledge-base/all-articles/customer-comms/internal-comms/) documentation. - Engineering, Product, Security, Customer Success, and Sales: What questions do you anticipate they will have? What is it important that they know? - Support: What guidance will the support team need to handle customer tickets related to this change? - Once you receive approval to proceed with the breaking change you should create a [Support Preparedness issue](https://gitlab.com/gitlab-com/support/support-team-meta/-/blob/master/.gitlab/issue_templates/Support%20Preparedness.md?ref_type=heads) with instructions on how to handle customer queries and concerns. ### Customer Communication Please consult the [Customer Comms Handbook for R&D](https://internal.gitlab.com/handbook/engineering/r-and-d-pmo/knowledge-base/all-articles/customer-comms/). Specifically the sections on [Picking a communication method](https://internal.gitlab.com/handbook/engineering/r-and-d-pmo/knowledge-base/all-articles/customer-comms/method/) and [Approvals Needed](https://internal.gitlab.com/handbook/engineering/r-and-d-pmo/knowledge-base/all-articles/customer-comms/risk-levels/) What is your customer communication strategy? ## Approval Process You should ensure you have received approval and communicated your breaking change at least **three months** in advance of the implementation date. It is recommended that you open this request at least **six months** in advance to have time for the approval process. It is the requestor's responsibility to drive the approval process forward and ensure it is completed in the recommended timeframe. - [x] Tag in the appropriate Product Director and Engineering Director for your product area. They may ask for further clarifications. - [x] Approved by **Product Director or GPM**: @jordanjanes - [x] Approved by **Engineering Director**: @mnohr - [x] Once you have approval from both Product GPM or Director & Engineering Director, add the label "Breaking Change: Awaiting Support Approval" and remove the prior label. - [x] Share this issue in the [#support_leadership](https://gitlab.enterprise.slack.com/archives/C01F9S37AKT) slack channel and ask for a review from the support perspective. Once you get signoff from 1 Support Senior Leader, add the label "Breaking Change: Awaiting VP Approval" and remove the prior label. - [ ] Approved by **Support Stable Counterpart (If Applicable)**: \< Add Name \> - [x] Approved by **Support Senior Manager or Director**: @leeeee-gtlb - [x] Tag in the appropriate Product and Engineering VPs or Senior Directors for your product area. They may ask for further clarifications. - [ ] Approved by **Product VP or Senior Director**: \< Add Name \> - [x] Approved by **Engineering VP or Senior Director**: @timzallmann - [x] Once you have approval from both the Engineering & Product VP or Senior Director, add the "Breaking Change: VP Approved" label. **Please keep this issue open** until the breaking change has been implemented. This ensures all breaking changes are visible to leadership on [this board.](https://gitlab.com/gitlab-com/Product/-/boards/9741901) - [x] Once you have VP approval, create a [public deprecation issue](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/issue_templates/Deprecations.md) that will serve as a source of truth for customers in regards to the change. - [x] Ensure the change is added to the deprecations docs page by following [this guidance.](https://docs.gitlab.com/development/deprecation_guidelines/#update-the-deprecations-and-removals-documentation) - [ ] Seek approval on any customer comms from the **Corporate Communications** and **Customer Success** teams