Draft: Separate unit primitives per duo chat tool
What does this MR do and why?
This MR addresses Separate unit primitives per duo chat tool (#468829), which effectively addresses the following issues:
- https://gitlab.com/gitlab-org/gitlab/-/issues/456703+
- Refactor AIGateway::Client (and the other servi... (#462694)
- !156222 (comment 1964948955)
- Add changes to accommodate slash commands unit ... (!156792 - merged)
- gitlab-org/modelops/applied-ml/code-suggestions/ai-assist!973 (comment 1961285748)
This also partially covers Draft: POC: Local docs search with bm25 (gitlab-org/modelops/applied-ml/code-suggestions/ai-assist!974 - closed) that we can get embeddings from self-managed models via LiteLLM client, which later can be fed to self-managed ElasticSearch/OpenSearch for semantic search.
Related MRs:
- Draft: Lite LLM proxy endpoint (gitlab-org/modelops/applied-ml/code-suggestions/ai-assist!995 - closed)
- https://gitlab.com/gitlab-org/customers-gitlab-com/-/merge_requests/10316+
Keep the boundary of the classes/modules responsibilities
One of the goals of this MR is to fix the boundary of the classes/modules responsibilities in order to avoid spaghetti code pattern.
The
AiGateway::Client
as of today is in a bad shape that it doesn't correctly reflect the interfaces of AI Gateway. It's tailored only for/v1/chat/agent
and now it's spilling the business logic asendpoint_url
parameter. For instance,::Gitlab::Llm::AiGateway::Client::DEFAULT_INSTANT_MODEL
andGitlab::Llm::AiGateway::Client::ConnectionError
doesn't make sense at all as it varies per feature. This is something we need to follow-up to correct the client interface.From client perspective, their concern is which services, models or features to access, and low-level stuff like authentication and connection protocols should be abstracted under the client libraries. Like demonstrated in typical client libraries (e.g. Google Cloud libraries) and
AiGateway::DocsClient
, we should introduce a new client per service. This means we should haveDuoChatClient
,DocsSearchClient
,VertexAIProxyClient
,AnthrpicProxyClient
etc instead of the singleAiGateway::Client
class. It should be relatively straight-forward to have the shared module for handling AI Gateway specific low-level stuff and plug it into these clients. Imagine, as a client, you would want to use a "BigQuery" service to create a dataset, not "Google Cloud Platform" to create a dataset (As that's too broad, you wouldn't have idea in which service the dataset will be created).
ref: #462694
MR acceptance checklist
Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
Screenshots or screen recordings
Screenshots are required for UI changes, and strongly recommended for all other merge requests.
Before | After |
---|---|
How to set up and validate locally
Numbered steps to set up and validate the change are strongly suggested.