**DRIs: `@mkaeppler` (Cloud Connector), `@shinya.maeda` (AI Framework)** Documentation has been published here: https://docs.gitlab.com/ee/user/gitlab_duo_chat.html#for-self-managed-users ## Problem GitLab Duo (chat) is currently available only for SaaS customers. This epic aims to bring GitLab chat functionality to self-managed customers via GitLab cloud connector. Similar to code suggestions, this will require an active internet connection between the self-managed instance and GitLab's Cloud Infrastructure. In %"16.4", we will perform some technical exploration to understand how to support GitLab chat for self-managed customers and identify a list of open questions that need to be worked through and clarified. Note: Dedicated is the same as SM. ### Technical proposal Fundamentally, our approach to delivering Duo Chat to SM/Dedicated will be similar to how we solved [Code Suggestions](https://docs.gitlab.com/ee/development/cloud_connector/code_suggestions_for_sm.html). Because the problem is similar (stateless conversation between gitlab-rails and an AI model), the solution is similar too. However, there are notable differences too, for example the necessity to query a vector store, which will have to be made available for self-managed. The following sections therefore break down the work required into problems that have known, existing solutions and those with less certainty that will require additional exploration. #### Problems that have existing solutions - **3P model calls will move from gitlab-rails to the AI gateway.** Chat requests and responses are currently scoped entirely to gitlab-rails, with the respective Ruby code modules talking directly to 3P providers like Anthropic. We need to push these calls down into the AI gateway, so that self-managed customers can benefit from the abstraction and access credential management provides by this system. We can do this in two different ways: 1. By first using [provider-specific endpoints](https://docs.gitlab.com/ee/architecture/blueprints/ai_gateway/index.html#exposing-ai-providers) rather than a `stable API`, which will allow a low friction transition in gitlab-rails because we only need to swap out the endpoint. _This is only a valid approach to take pre-GA._ 2. By defining a stable API endpoint for Chat. This is a GA requirement; however, it complicates the implementation because the interface contract with Rails would change. Either way, the following work items would result from that: - Build Anthropic Chat endpoint(s) in AI gateway - Provide Anthropic credentials via Runway vault - Update gitlab-rails `Chat` implementation to call into AI gateway instead - **Authentication flow: no change.** We will generally re-use the authentication solution we built for Code Suggestions. However, there is some exploratory work to do around how to map JWTs to Cloud Connector features (see below). The resulting architecture would look as follows: ```plantuml @startuml node "Client services" { [IDE/Web UI] as IDE } node "GitLab Inc. services" { [AI gateway] as AI [GitLab] as SAAS } node "Self-managed services" { [GitLab] as SM } node "Third-party services" { package "3P models" as MOD { [Anthropic] [Vertex AI] } } [IDE] --> SAAS : " chat input" [IDE] --> SM : " chat input" SAAS ->SAAS : "enrich prompt" SM -->SM : "enrich prompt" SM -> AI : " enriched request" SAAS --> AI : " enriched request" AI --> MOD : " model-specific request" @enduml ``` #### Problems that require new solutions - **DESCOPED: Handling GitLab Docs embeddings.** We already have an `embeddings` Postgres database for GitLab Docs that is in use by SaaS. This database cannot currently be used by self-managed. We need to find a solution to letting self-managed instances query embeddings for the version of GitLab Docs specific to their instance. - Currently being fleshed out in https://gitlab.com/gitlab-org/gitlab/-/issues/424849 - Update 2023-11-03: We have descoped embeddings from the initial Self Managed iteration. This mean documentation search would not work. - **Open: Mapping JWT token scope to features.** We only a single Cloud Connector feature so far: Code Suggestions. This was a simple case too because there was a 1-to-1 mapping from feature to system at the time. This is not the case anymore: with the AI gateway now hosting two and soon more AI features, we may need to rework how we map tokens (and the features scopes they represent) to specific AI gateway endpoints. - Currently being fleshed out in https://gitlab.com/gitlab-org/gitlab/-/issues/425417 - Update 2023-11-03: This is now solved, see above issue ### Steps | Theme | Team | Issue/Epic/Description | Target Milestone | Status | |-------|------|------------------------|------------------|--------| | GitLab Duo (chat) for SM | Cloud Connector | https://gitlab.com/groups/gitlab-org/-/epics/11546+ | 16.6 | Implementation complete, documentation in progress | | GitLab Duo (chat) for SM | Cloud Connector | https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist/-/issues/289+s | 16.6 | Done | | GitLab Duo (chat) for SM | Cloud Connector | https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist/-/issues/317+s | 16.6 | Done | | GitLab Duo (chat) for SM | AI Framework | https://gitlab.com/gitlab-org/gitlab/-/issues/431563+s | 16.7 | Done | | GitLab Duo (chat) for SM | AI Framework | https://gitlab.com/gitlab-org/gitlab/-/issues/434916+s | 16.7 | Done | | GitLab Duo (chat) for SM | AI Framework | https://gitlab.com/gitlab-org/gitlab/-/issues/433213+s | 16.7 | Done | | GitLab Duo (chat) for SM | AI Framework | https://gitlab.com/gitlab-org/gitlab/-/issues/434925+s | 16.7 | Done | | GitLab Duo (chat) for SM | AI Framework | https://gitlab.com/gitlab-org/gitlab/-/issues/435911+s | 16.8 | Done | | GitLab Duo (chat) for SM | AI Framework | https://gitlab.com/gitlab-org/gitlab/-/issues/433470+s | 16.8 | Done | | GitLab Duo (chat) for SM | AI Framework | https://gitlab.com/gitlab-org/gitlab/-/issues/425917+s | 16.8 | Done | | GitLab Duo (chat) for SM | AI Framework | https://gitlab.com/gitlab-org/gitlab/-/issues/434164+s | 16.8 | Done | | GitLab Duo (chat) for SM | AI Framework | https://gitlab.com/gitlab-org/gitlab-vscode-extension/-/issues/1199+s | 16.8 | Done | | GitLab Duo (chat) for SM | AI Framework | https://gitlab.com/gitlab-org/gitlab/-/issues/434802+s | 16.8 | Done | | GitLab Duo (chat) for SM | AI Framework | https://gitlab.com/gitlab-org/gitlab/-/issues/434670+s | 16.8 | Done | | GitLab Duo (chat) for SM | AI Framework | https://gitlab.com/gitlab-org/gitlab/-/issues/434923+s | 16.8 | Done | | GitLab Duo (chat) for SM | AI Framework | https://gitlab.com/gitlab-org/gitlab/-/issues/434198+s | 16.8 | Done | | GitLab Duo (chat) for SM | AI Framework | https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist/-/issues/371+s | 16.8 | Done | | GitLab Duo (chat) for SM | Cloud Connector | https://gitlab.com/gitlab-org/gitlab/-/issues/437139+s | 16.8 | Done | | GitLab Duo (chat) for SM | Cloud Connector | https://gitlab.com/gitlab-org/cloud-connector-team/decisions/-/issues/1+s | 16.8 | Done | | GitLab Duo (chat) for SM | Cloud Connector | https://gitlab.com/gitlab-org/gitlab/-/issues/425047+s | 16.8 | Done | | GitLab Duo (chat) for SM | Cloud Connector | Instrument the number unique instances using GitLab Chat on SM. Product analysis team does not have capacity for the remainder of this year. For the initial iteration, data needs to be piped through to Snowflake but visualization can be in Kibana. | 16.8 | Done. `X-Gitlab-Realm` is sent from `AiGateway::Client`. See https://gitlab.com/groups/gitlab-org/-/epics/11251#note_1700271865 for more details. | Post-release: | Theme | Team | Issue/Epic/Description | Target Milestone | Status | |-------|------|------------------------|------------------|--------| | GitLab Duo (chat) for SM | AI Framework | https://gitlab.com/gitlab-org/gitlab/-/issues/435820+s | 16.9 | In refinement | | GitLab Duo (chat) for SM | Cloud Connector | https://gitlab.com/gitlab-org/gitlab/-/issues/435847+s | 16.9 | Done | | GitLab Duo (chat) for SM | AI Framework | https://gitlab.com/gitlab-org/gitlab/-/issues/434927+s | 16.9 | Not started (Optional) | | GitLab Duo (chat) for SM | AI Framework | https://gitlab.com/gitlab-org/gitlab/-/issues/435837+s | 16.9 | Not started (Optional) | | GitLab Duo (chat) for SM | Cloud Connector | https://gitlab.com/gitlab-org/gitlab/-/issues/438405+s | 16.9 | Not started | | GitLab Duo (chat) for SM | AI Framework | https://gitlab.com/gitlab-org/gitlab/-/issues/435292+s | 16.9 | In Progress | | GitLab Duo (chat) for SM | ? | https://gitlab.com/gitlab-org/customers-gitlab-com/-/issues/8442+s | by 16.10 | Not started | ### Open Questions * How can instance admins enable GitLab chat? Explored in https://gitlab.com/gitlab-org/gitlab/-/issues/424642 * Will we need a separate legal/privacy acknowledgement checkbox from code suggestions, or can all GitLab Cloud Connector features share the same privacy disclaimer / checkbox? * **A:** It is part of the `AI powered` checkbox * Do self-managed / Dedicated customers need their own accounts with 3P service providers? Or can we integrate everything into the same platform account currently used by SaaS? - **A:**\* No, they don't need dedicated accounts. They will use the AI gateway instead, as they do for Code Suggestions. * Is Chat entirely run off of 3P AI models? - **A:** Yes, we do not use GitLab models for Chat. We currently target Anthropic and are in the process of supporting VertexAI. * [Will Chat history work for self-managed right out gate?](https://gitlab.com/gitlab-org/gitlab/-/issues/425917 "Check whether Chat history will work as-is for SM") * **A:** Yes, verified to work as well for SM. ### Related - https://gitlab.com/groups/gitlab-org/-/epics/10585