This epic is intended to capture the existing plan and iterative cadence for Custom Model support to self-hosted (open source model-based) Duo Chat. Our iteration will follow a data-driven flow, predicated upon: * identifying open source, licensable models fit for the Chat use cases * validating and baselining performance of our selected open source model against the Chat use cases * prompt iteration and testing Dot.com Chat is currently GA, but will continue to incorporate as tools additional Duo features that are currently standalone. Self-Hosted support to additional tool features in Chat will follow GA of those features. Work related to enabling self-hosted Duo Chat GA has been pulled out into a separate epic [here](https://gitlab.com/groups/gitlab-org/-/epics/15227), but will be added back to this base epic after GA. ## Resources - https://gitlab.com/groups/gitlab-org/-/epics/13297+ - [Duo Chat development guide](https://docs.gitlab.com/ee/development/ai_features/duo_chat.html) - [Chat prompts in AIG](https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist/-/tree/main/ai_gateway/prompts/definitions/chat?ref_type=heads) - [Chat prompts in Rails](https://gitlab.com/gitlab-org/gitlab/-/tree/master/ee/lib/gitlab/llm/chain/tools) - [Context fetching tools in AIG](https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist/blob/main/ai_gateway/chat/tools/gitlab.py#L101-101) - [Self-Hosted Chat Validation Results](https://docs.google.com/spreadsheets/d/1qdVO1yQhFPIzRzRrHzDVcu3aNGDgXTVaYRlYJH-0K0A/edit?gid=0#gid=0) # Background There are two main modalities by which Chat delivers information to users. 1. agentic (within the Chat Window) 2. single action executor (slash commands or function buttons) ##### Agentic When users input questions or requests into the Duo Chat box, a zero-shot agent is used to determine the routing of the user's input. If the user's question is related to code generation, then the user's input is sent unaltered to Claude to be directly answered. If the user's input is NOT related to code generation, the zero-shot agent directs it as appropriate to one or many of a series of specialized prompts (aka [tools](https://gitlab.com/gitlab-org/gitlab/-/tree/master/ee/lib/gitlab/llm/chain/tools)). There are also additional non-prompt based .rb [chain](https://gitlab.com/gitlab-org/gitlab/-/tree/master/ee/lib/gitlab/llm/chain) files that enable additional functions such as context fetching, /command, streaming, etc. * [**Zero Shot Agent**](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/gitlab/llm/chain/agents/zero_shot/executor.rb): this prompt defines for the LLM the tools it may choose from (defined below), directs a ReAct Chain-of-Thought, parses the final LLM response, defines default error messages, sets resource limits (10% of total limit). * [**Code Generation**](https://gitlab.com/gitlab-org/gitlab/-/issues/474229): **this tool does not currently exist in the gitlab.com version of Duo Chat**, as code generation prompts are currently passed by the zero-shot agent directly to the foundational model. * The zero-shot agent and the tool are being[ moved to the AI Gateway](https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist/blob/main/ai_gateway/chat/tools/gitlab.py). * [**EpicReader**](https://gitlab.com/gitlab-org/gitlab/-/tree/master/ee/lib/gitlab/llm/chain/tools/epic_reader): Useful tool when you need to retrieve information about a specific epic. In this context, word `epic` means high-level building block in GitLab that encapsulates high-level plans and discussions. Epic can contain multiple issues. Action Input for this tool should be the original question or epic identifier. * the EpicReader prompt directs the LLM to fetch information about a public Epic based on either the user's current page \[current\] or a provided \[iid\], \[url\], or \[reference\]. The information on the referenced Epic is then added into the prompt to handle questions. As self-hosted model version would likely need a different way to provide the Epic content as a prompt injection rather than as an integrated information retrieval task * [**IssueReader**](https://gitlab.com/gitlab-org/gitlab/-/tree/master/ee/lib/gitlab/llm/chain/tools/issue_reader): Gets the content of the current issue (also referenced as this or that) the user sees or a specific issue identified by an ID or a URL. In this context, word `issue` means core building block in GitLab that enable collaboration, discussions, planning and tracking of work. Action Input for this tool should be the original question or issue identifier. * similar to EpicReader, IssueReader prompt directs the LLM to fetch information about a public Epic based on either the user's current page \[current\] or a provided \[iid\], \[url\], or \[reference\]. The information on the referenced Epic is then added into the prompt to answer the question. * [**GitlabDocumentation**](https://gitlab.com/gitlab-org/gitlab/-/tree/master/ee/lib/gitlab/llm/chain/tools/gitlab_documentation): This tool is beneficial when you need to answer questions concerning GitLab and its features. Questions can be about GitLab's projects, groups, issues, merge requests, epics, milestones, labels, CI/CD pipelines, git repositories, and more. * **GitlabDocumentation** uses TanukiBot, which leverages the GL documentation embedding db to return relevant documentation for inclusion in the prompt. * [**MergeRequestReader**:](https://gitlab.com/gitlab-org/gitlab/-/tree/master/ee/lib/gitlab/llm/chain/tools/merge_request_reader) gets the content of the current MR - This feature is not yet fully rolled out. * [**~~CIEditorAssistant~~**](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/gitlab/llm/chain/tools/ci_editor_assistant/executor.rb)~~: Useful tool when you need to provide suggestions regarding anything related to ".gitlab-ci.yml" file. It helps with questions related to deployments, configuring CI/CD pipelines, defining CI jobs, or environments. It can not help with writing code in general or questions about software developmentSingle Action Executor~~ CIEditorAssistant was removed For discrete actions that don't require an agentic flow, the [single_action_executor.rb](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/gitlab/llm/chain/agents/single_action_executor.rb) is used as the prompt. This supports [slash commands](https://docs.gitlab.com/ee/user/gitlab_duo_chat/examples.html#supported-slash-commands) as well as buttons that exist in the GitLab UI external to Chat but have responses returned via Chat. * [**Explain Code**](https://gitlab.com/gitlab-org/gitlab/-/tree/master/ee/lib/gitlab/llm/chain/tools/explain_code): Accessed using **/explain**. This is useful tool to explain code snippets and blocks. * Explain Code injects into the prompt a code snippet or block that the user has selected * [**RefactorCode**](https://gitlab.com/gitlab-org/gitlab/-/tree/master/ee/lib/gitlab/llm/chain/tools/refactor_code): Accessed using **/refactor**. This is a useful tool to refactor source code. * RefactorCode injects into the prompt code that the user has selected and then prompts the LLM to refactor it. * [**FixCode**](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/gitlab/llm/chain/tools/fix_code/executor.rb): Accessed using **/fix**. Useful tool to fix source code. * [**WriteTests**](https://gitlab.com/gitlab-org/gitlab/-/tree/master/ee/lib/gitlab/llm/chain/tools/write_tests): Accessed using **/test**. This is a useful tool to write tests for source code. * WriteTests injects into the prompt code that the user has selected and then prompts the LLM to write a test * [**Explain Vulnerability**](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/gitlab/llm/chain/tools/explain_vulnerability/executor.rb): Accessed via the 'Explain or Resolve with AI' button on a[ static vulnerability page](https://gitlab.com/gitlab-examples/wayne-enterprises/wayne-financial/simple-notes-demo/-/security/vulnerabilities/66778996) or with **/vulnerability_explain**. This tool explains vulnerability in code. * [**Troubleshoot_Job**](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/gitlab/llm/chain/tools/troubleshoot_job/executor.rb): also know as Root Cause Analysis (RCA). Accessed via the 'Troubleshoot; button on a failed pipeline or with **/troubleshoot**. Useful tool to troubleshoot job-related issues * [**SummarizeComments**](https://gitlab.com/gitlab-org/gitlab/-/tree/master/ee/lib/gitlab/llm/chain/tools/summarize_comments): Accessed via the 'View Summary' button on an issue. This tool is useful when you need to create a summary of all notes, comments or discussions on a given, identified resource. * SummarizeComments directs the LLM to use the IssueReader tool to identify the issue to be summarized, and then prompts the LLM for summarization.  ### Challenges Given the wide breadth of Chat's functionalities, it may take numerous OS models to emulate the functionality of Chat that large proprietary models such as Anthropic can provide. A mixture of experts ensemble will therefore be a likely path forward. For example, while Mixtral and CodeGemma perform well on Validation baselines for code generation, they perform poorly on other Chat datasets such as issue/epic and code explanation. We will need to identify and validate other open source models that handle NLP tasks, as well as RAG injections and so on. Emulating Chat functionality may further require the construction of numerous use-case specific prompts (tools), as well as the ability to correctly route user's inputs to those tools as appropriate. # Validation/Quality Results * [Self-Hosted Chat Validation Results](https://docs.google.com/spreadsheets/d/1qdVO1yQhFPIzRzRrHzDVcu3aNGDgXTVaYRlYJH-0K0A/edit?gid=0#gid=0)