RAG MVP for Chat - Documentation

The first iteration of RAG customization for a self-hosted feature will begin with documentation q&as within Duo Chat. Duo Chat is currently configured to enable question answering on GitLab documentation. We can offer customizations that would allow a user to ask and answer questions on their own documentation.

Pre-Conditions

User is hosting ElasticSearch that can be configured with a self-managed GitLab instance for indexing/embedding

user is self-hosting a LLM to support Chat functionality

Self-Hosted Chat Development Steps

• Instance-Level Configuration for External Documentation
•   Configuration option to select model (#454323 - closed) • Igor Drozdov, Patrick Cyiza • 17.1 • On track
•   Prompt Routing in GitLab Rails (#455311 - closed) • Patrick Cyiza • 17.1 • On track
•   GitLab Duo License Management for Self-Hosted M... (#455858 - closed) • Igor Drozdov • 17.2 • Needs attention
• Baseline OS models for comparison
  • Validate and baseline Mixtral 7x8B
    • for Code Generation Mixtral 8x7B performs very well, outperforming Claude 2.1
  • Validate and baseline Code Gemma
    • for Code Completion, Gemma performs well
•   Mixtral 8x7B-Instruct/8x22b prompt: IDE Code Ge... (#455303 - closed)
•   Mistral 7B prompt: IDE Code Generation (#460068 - closed)
•   AI Gateway API endpoint routing (#455315 - closed)
•   Self-Hosted MVP: Send configured API key to AI ... (#463760 - closed)
•   Document installing AI Gateway via Docker (#452489 - closed)
•   Produce short videos for self-hosted models MVP (#463821 - closed)

Customization Flow

• customer configures their self managed GitLab instance to connect to a self-hosted embedding/indexing model (ie ElasticSearch)
• customer provides routing for documentation(s) that serve as context for question answering
• within the GitLab UI, the customer can configure indexing/chunking approach
• the index(s) are stored within their Gitlab instance
• the customer can configure hotkeys (ie @companyx) to indicate when they want to invoke their own documentation
• the customer can ask questions to Duo chat using an @ hotkey to invoke specific documentation