_Legacy title: Filter Chat tools on AI Gateway_ ## Current status - The [feature flag](https://gitlab.com/gitlab-org/gitlab/-/issues/466910) is [enabled by default](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/feature_flags/beta/v2_chat_agent_integration.yml). It will be released in %17.5. - The feature flag is enabled globally on gitlab.com. - The feature flag _might_ be [selectively disabled on some users](https://gitlab.com/gitlab-org/gitlab/-/issues/466910#how-to-disable-the-feature-flag-for-a-specific-user). To see the list, run `/chatops run feature get v2_chat_agent_integration_override` on `#production` Slack channel. - See [Known issues](https://gitlab.com/groups/gitlab-org/-/epics/13533#known-issues-that-duo-chat-team-is-actively-fixing) or [Runbook](https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/duo-chat/README.md?ref_type=heads) for more errors, logs and metrics. - To see [Expanded AI logging](https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/duo-chat/README.md?ref_type=heads#expanded-ai-logging) enabled users, run `/chatops run feature get expanded_ai_logging` on `#production` Slack channel. ## Report a bug NOTE: See [Known issues](https://gitlab.com/groups/gitlab-org/-/epics/13533#known-issues-that-duo-chat-team-is-actively-fixing) section before filing a report. The other users might have reported the same issue. If you find a bug or unexpected behavior, please create a new issue with the following information: - User name who encountered the bug (e.g. `@shinya.maeda`) - When it happened (e.g. Around 2024/09/16 01:00 UTC) - Steps to reproduce (e.g. 1. Ask a question "xxx" 2. Click ...) - Whether the bug can be reproduced with Chat Agent V1 as well. See [how to disable the feature flag for a specific user](https://gitlab.com/gitlab-org/gitlab/-/issues/466910#how-to-disable-the-feature-flag-for-a-specific-user). - A link to a Slack discussion, if any. And ping `@gitlab-org/ai-powered/duo-chat` members in the created issue. We will investigate further. Ref: [Before start investigation](https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/duo-chat/README.md?ref_type=heads#before-start-investigation). ## Timeline <details> <summary>past events</summary> - The [feature flag](https://gitlab.com/gitlab-org/gitlab/-/issues/466910) is enabled on GitLab team members between 2024/09/19 05:00 UTC - 12:00 UTC (7 hours). => Done. - The [feature flag](https://gitlab.com/gitlab-org/gitlab/-/issues/466910) is enabled on GitLab team members between 2024/09/20 02:00 UTC - 14:00 UTC (12 hours). => Done. - The feature flag is permanently enabled on Duo Chat team members and [some volunteers](https://gitlab.com/gitlab-org/gitlab/-/issues/484755). - 2024/09/23 Mon: Permanently enable the feature flag to GitLab team members on gitlab.com. - 2024/09/24 Tue: Roll out the feature flag to 25% of users on gitlab.com. - 2024/09/26 Thu: Roll out the feature flag to 50% of users on gitlab.com. - 2024/09/27 Fri: Roll out the feature flag to 75% of users on gitlab.com. - 2024/09/28 Sat-Sun: Roll back the feature flag to 0% of users on gitlab.com derisk incidents in weekend. - 2024/09/30 Mon ~ 2024/10/04 Fri: - Roll out the feature flag to 100% of users on gitlab.com. - Make sure that no critical issues are raised during the period. </details> - 2024/10/07 Mon: - Enable the flag by default on source code. => [Done](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/168326) ## Known issues that Duo Chat team is actively fixing - https://gitlab.com/gitlab-org/gitlab/-/issues/493507+ ## DRIs - @brytannia - @shinya.maeda ## Why this is important Because this is blocker against https://gitlab.com/groups/gitlab-org/-/work_items/14735+ %17.4 release. Injected Context is supported in Chat Agent V2, only. ## Problem Duo Chat is a cloud feature and it depends on 3d party providers (currently Anthropic and Google). If users gain access to Chat without paying, GitLab will face active losses, and we should prevent this. In context of https://gitlab.com/groups/gitlab-org/-/epics/12556+ we need to be able limit user access to AI features, based on the addon they purchased. In context of Chat such features could be the [Tools](https://gitlab.com/gitlab-org/gitlab/-/blob/2015f9e0f8f402cb43ad4cef2850aa398d0786b2/ee/lib/gitlab/llm/chain/tools/tool.rb#L12-12) that are used to get the response. ## Proposal How we get to this solution: 1. We can't trust Rails monolith, as users can temper with code on SM instances and try to gain assess to our cloud features without paying for it. 2. We might want to block assess to a chat feature retroactively: if user is staying on an outdated version of the GitLab, but we already moved a feature to a new addon that user haven't paid for, we want to close an access. 3. We need to make a decision if user has an access to the chat feature on an environment that is controlled by GitLab, in our case it's Gateway. That's why GW needs to be able to reason about chat features (tools). 4. We can do it by either filtering the LLM response, or assembling the initial prompt that contains the list of all the available tools. 5. After long discussions the first option was dismissed because it would come with several disadvantages [1](https://gitlab.com/gitlab-org/gitlab/-/issues/439535#note_1792187658 "Discovery Spike: Subdividing Chat Feature Based on User's Purchased Addon") [2](https://gitlab.com/gitlab-org/gitlab/-/issues/446206#note_1804453220 "Feature proposal: Filter LLM response on Gateway") 6. It left us with prompt assembling on GW. We assemble prompt on Gateway: 1. Rails sends the question and additional context to the Gateway 2. Gateway get information from Cloud connector service on Chat tools (Unit primitives) available to user. 3. Based on this information Gateway assembles the prompt. 4. Prompt is sent to LLM. 5. Gateway gets response, process it and stream the answer to the Rails side. The answer can contain either name of the selected tool or chunks of the final answer. 6. Rails receives the answer and if it's a tool and not the final answer, it executes tool. 7. Tool might perform the request to LLM via Gateway using an endpoint different from the one used on the first step. 8. Gateway received the request and verifies that user has an access to the tool (similar to the step 2) and sends the prompt to LLM to execute. 9. Rails tool received the answer if the answer isn't final, we returns to the step 6 and send another request to the Gateway including initial question, answers from the Gateway and additional context retrieved by Rails tools. 10. Gateway sends request back to LLM and steps 5-9 are repeated until we get the final answer that is streamed to the user in the chat. 11. The final answer and the initial question are saved to the chat history.  ## Workflow breakdown This workload breakdown illustrates what tasks can be done in parallel and what are the blockers. 1. Phase 1 "Foundation". We create [an endpoint](https://gitlab.com/gitlab-org/gitlab/-/issues/452204) in the Gateway that can assemble Chat ReAct prompt and [stream](https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist/-/issues/468) answers back. 2. Phase 2 "Integration". We [integrate](https://gitlab.com/gitlab-org/gitlab/-/issues/456258) a new endpoint in Rails and add capability to [filter prompt](https://gitlab.com/gitlab-org/gitlab/-/issues/456259) on Gateway. At this point Chat will be able to answer same questions via new endpoint on Gateway. In parallel, Gateway is integrated with Cloud connector service that allow us to manipulate with prompts. 3. Phase 3 "Expansion". We add an authorisation layer to requests that executed on Gateway for Chat tools. Chat allows to execute [slash commands](https://gitlab.com/gitlab-org/gitlab/-/issues/456703) that are trigger Tools directly, we need to be able prevent users from using those commands if they don't have required add-on. For that we use the capabilities that were added in the previous phase. 4. Phase 4 "Migration". While we were working on previous phases of this project, Chat gets more improvements that are merged into Rails codebase. Last step will be to [port final prompts](https://gitlab.com/gitlab-org/gitlab/-/issues/463383) from Rails to the Gateway and ensure that quality of responses is the same. At this point adding prompt changes to Rails isn't allowed anymore, all merge requests should be happening to Gateway. Then we can enable feature flags. ## Testing results We can reuse existing Chat evaluation framework to make sure that the quality of chat responses is similar to what we have already. As CEF uses GitLab API to get chat completions, we will be able to test changes in the Gateway in the similar fashion as we test Rails changes. To evaluate correctness of the responses we need to have running Gateway and Rails server, which is the true for now as well. Only difference is that when this epic will be implemented, we'll be doing Chat ReAct prompt changes in Gateway, and not in Rails, but from the testing perspective, nothing is changed. All work should happens behind feature flags both in the Gateway and Rails monolith. Also, we need to implement [an additional endpoint](https://gitlab.com/gitlab-org/gitlab/-/issues/456260) for Gateway that will help with debug as it might be not obvious if user has or doesn't have an access to specific type of questions.