Skip to content

feat: anthropic proxy endpoint

Shinya Maeda requested to merge anthropic-proxy-endpoint into main

NOTE: This MR has high-priority due to the timeline.

What does this merge request do and why?

This MR implements Anthropic proxy endpoint according to the AI Gateway ADR 002: Exposing proxy endpoints to AI providers. The main goal of these endpoints is to enable the independent AI features in self-managed instances within the proposed timeline. See the issue and this issue for more information.

For GitLab-Rails counter-part, see Anthropic client requests via AI Gateway proxy (gitlab-org/gitlab!150255 - closed)

How to set up and validate locally

  1. poetry run ai_gateway. Enable AIGW_AUTH__BYPASS_EXTERNAL for testing purpose.
  2. Execute the following curl CLI in your terminal:
curl -X 'POST' \
  'http://0.0.0.0:5052/v1/proxy/anthropic/v1/complete' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  --header "anthropic-version: 2023-06-01" \
  -d '{
    "model": "claude-2.1",
    "max_tokens_to_sample": 1024,
    "prompt": "\n\nHuman: Hello, Claude\n\nAssistant:"
}'
curl -X 'POST' \
  'http://0.0.0.0:5052/v1/proxy/anthropic/v1/messages' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  --header "anthropic-version: 2023-06-01" \
  -d '{
    "model": "claude-3-opus-20240229",
    "max_tokens": 1024,
    "messages": [
        {"role": "user", "content": "Can you sign a long song?"}
    ],
    "stream": true
}'

Merge request checklist

  • Tests added for new functionality. If not, please raise an issue to follow up.
  • Documentation added/updated, if needed.
Edited by Shinya Maeda

Merge request reports