Skip to content

Execute chat requests via new endpoint in AI Gateway

Tetiana Chupryna requested to merge 456258-new-chat-executor into master

What does this MR do and why?

This MR implements https://gitlab.com/gitlab-org/gitlab/-/issues/456258+

To be able filter prompts on AI Gateway, a new endpoint v2/chat/agent was created. This endpoint returns response in more structured was and also supports streaming with json_objects.

Because of significant changes needed to be done to parsing and processing answers, I decided to create a new chat agent SingleAction, this agent is similar to ZeroShot, but we don't do any processing of user question.

You can check the new AI Gateway endpoint via docs endpoint on your gdk.

In short, it accepts request with next body:

{
  "prompt": "string",
  "options": {
    "chat_history": "string",
    "agent_scratchpad": {
      "agent_type": "react",
      "steps": [
        {
          "thought": "string",
          "tool": "string",
          "tool_input": "string",
          "observation": "string"
        }
      ]
    },
    "context": {
      "type": "issue",
      "content": "string"
    }
  }
}

and returns answers in next format:

{
  "type":"action",
  "data":
  {  "thought":"Thought: The question is asking about the title of the current issue that the user is viewing. To get this information, I need to use the \"issue_reader\" tool.",
     "log":"Thought: Thought: The question is asking about the title of the current issue that the user is viewing. To get this information, I need to use the \"issue_reader\" tool.\n\nAction: issue_reader\nAction Input: what is the title of this issue",
     "tool":"issue_reader",
     "tool_input":"what is the title of this issue"
  }
}

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Screenshots or screen recordings

v2

Evaluation of chat

It will be a follow-up as we have disparity between Gateway and Rails prompts, so evaluation results won't be comparable

How to set up and validate locally

Enable v2_chat_agent_integration flag and chat should work the same as now.

Related to #456258

Edited by Tetiana Chupryna

Merge request reports