Skip to content

Support Claude 3 models for Duo Chat

Jessie Young requested to merge jy-claude-3 into main

What does this merge request do and why?

Update Duo Chat to use Anthropic Claude 3 Sonne... (gitlab-org/gitlab#444629 - closed)

Claude's new models require usage of the messages API, documented here: https://docs.anthropic.com/claude/reference/messages_post

We do not support the Messages API in the Gateway yet, related issue: Migrate to Anthropic Messages API (#419 - closed)

This MR adds support for the Messages API only when the new models are requested via a header. I tested it locally via http://localhost:5052/docs#/chat/chat_v1_chat_agent_post with "claude-3-opus-20240229" and "claude-3-sonnet-20240229" and "claude-3-haiku-20240307"

I did not add logic to support Claude 3 for Code Suggestions. There's a different issue for that: Upgrade to Claude 3 for Code Generation (gitlab-org/gitlab#444405 - closed)

How to set up and validate locally

Ensure AI Gateway is running locally and then run the following curl request:

curl -X 'POST' \
  'http://localhost:5052/v1/chat/agent' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "prompt_components": [
    {
      "type": "string",
      "metadata": {
        "source": "string",
        "version": "string"
      },
      "payload": {
        "content": [{ "role": "system", "content": "You are a Python engineer"}, { "role": "user", "content": "define a function that adds numbers together" }],
        "provider": "anthropic",
        "model": "claude-3-sonnet-20240229",
        "params": {
          "stop_sequences": [
            "\n\nHuman",
            "Observation:"
          ],
          "temperature": 0.2,
          "max_tokens_to_sample": 1024
        }
      }
    }
  ],
  "stream": false
}'

Merge request checklist

  • Tests added for new functionality. If not, please raise an issue to follow up.
  • Documentation added/updated, if needed.
Edited by Alexander Chueshev

Merge request reports