Skip to content

Draft: Separate unit primitives per duo chat tool

Shinya Maeda requested to merge separate-unit-prem-per-duo-chat-tool into master

What does this MR do and why?

This MR addresses Separate unit primitives per duo chat tool (#468829), which effectively addresses the following issues:

This also partially covers Draft: POC: Local docs search with bm25 (gitlab-org/modelops/applied-ml/code-suggestions/ai-assist!974 - closed) that we can get embeddings from self-managed models via LiteLLM client, which later can be fed to self-managed ElasticSearch/OpenSearch for semantic search.

Related MRs:

Keep the boundary of the classes/modules responsibilities

One of the goals of this MR is to fix the boundary of the classes/modules responsibilities in order to avoid spaghetti code pattern.

The AiGateway::Client as of today is in a bad shape that it doesn't correctly reflect the interfaces of AI Gateway. It's tailored only for /v1/chat/agent and now it's spilling the business logic as endpoint_url parameter. For instance, ::Gitlab::Llm::AiGateway::Client::DEFAULT_INSTANT_MODEL and Gitlab::Llm::AiGateway::Client::ConnectionError doesn't make sense at all as it varies per feature. This is something we need to follow-up to correct the client interface.

From client perspective, their concern is which services, models or features to access, and low-level stuff like authentication and connection protocols should be abstracted under the client libraries. Like demonstrated in typical client libraries (e.g. Google Cloud libraries) and AiGateway::DocsClient, we should introduce a new client per service. This means we should have DuoChatClient, DocsSearchClient, VertexAIProxyClient, AnthrpicProxyClient etc instead of the single AiGateway::Client class. It should be relatively straight-forward to have the shared module for handling AI Gateway specific low-level stuff and plug it into these clients. Imagine, as a client, you would want to use a "BigQuery" service to create a dataset, not "Google Cloud Platform" to create a dataset (As that's too broad, you wouldn't have idea in which service the dataset will be created).

ref: #462694

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

Before After

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

Edited by Shinya Maeda

Merge request reports