Skip to content

Support more Anthropic models in v1/chat/agent endpoint

Shinya Maeda requested to merge support-more-claude-models into main

What does this merge request do and why?

This MR adds support for claude-instant-1 and claude-instant-1.1 to the v1/chat/agent endpoint.

While I was debugging Pass Anthropic parameters to AI Gateway (gitlab-org/gitlab!139499 - merged), I realized that some of the Duo Chat logic still depends on these legacy models. If we don't support this in AI Gateway, GitLab-Rails raises a validation error on the model field.

This is blocker against Pass Anthropic parameters to AI Gateway (gitlab-org/gitlab!139499 - merged)

How to set up and validate locally

Requesting to the AI Gateway from the GitLab-Rails

user = User.first
options = { model: 'claude-instant-1', temperature: 0.5, max_tokens_to_sample: 1024, stop_sequences: ["\n\nHuman", "Test:"]}
Gitlab::Llm::AiGateway::Client.new(user).stream(prompt: "\n\nHuman: Can you sing a song?\n\nAssistant:", **options)

Confirm that the claude-instant-1 is passed to the Anthropic client:

{
    "correlation_id": "25849b398c6e402f80f9ec2d6082c932",
    "logger": "anthropic._base_client",
    "level": "debug",
    "type": "mlops",
    "stage": "main",
    "timestamp": "2023-12-13T05:16:10.106478Z",
    "message": "Request options: {'method': 'post', 'url': '/v1/complete', 'timeout': Timeout(connect=5.0, read=30.0, write=30.0, pool=30.0), 'files': None, 'json_data': {'max_tokens_to_sample': 1024, 'model': 'claude-instant-1', 'prompt': '\\n\\nHuman: Can you sing a song?\\n\\nAssistant:', 'stop_sequences': ['\\n\\nHuman', 'Test:'], 'stream': True, 'temperature': 0.5}}"
}

Merge request checklist

  • Tests added for new functionality. If not, please raise an issue to follow up.
  • Documentation added/updated, if needed.
Edited by Shinya Maeda

Merge request reports