Skip to content

Create an anthropic proxy endpoint in the AI gateway

Roy Zwambag requested to merge 289-anthropic-ai-gateway into main

What does this merge request do and why?

This MR creates a proxy endpoint that forwards the requests to anthropic. The only change it makes is converting the prompt to a format that anthropic expects (Human: {prompt}\nAssistant: {answer}). It sets some defaults, but all can be overridden with parameters.

How to set up and validate locally

Set ANTHROPIC_API_KEY ENV to your Anthropic API key. Make a call to anthropic

curl -X 'POST' \
  'http://0.0.0.0:5052/v2/anthropic' \
  -H 'Accept: application/json' \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer <access_token>' \
  -H 'X-Gitlab-Authentication-Type: oidc' \
  -d '{
  "prompt": "your prompt here",
  "model": "claude-2",
  "max_tokens_to_sample": 2048,
  "stop_sequences": [],
  "stream": false,
  "temperature": 0.1
}'

Merge request checklist

  • Tests added for new functionality. If not, please raise an issue to follow up.
  • Documentation added/updated, if needed.

Closes #289 (closed)

Edited by Roy Zwambag

Merge request reports