Skip to content

Use Anthropic proxy endpoints in Anthropic::Client

Shinya Maeda requested to merge make-anthropic-ai-client-compatible-with-ai-gateway into master

NOTE: This is a high-priority MR for the deadline.

NOTE: This MR is on top of Use Vertex AI proxy endpoints in VertexAI::Client (!153215 - merged).

What does this merge request do and why?

This MR uses AI Gateway's Anthropic AI proxy endpoints in Anthropic::Client. See AI Gateway ADR 002: Exposing proxy endpoints to AI providers for the overview of the changes.

This change is behind use_ai_gateway_proxy feature flag, which is disabled by default.

The main goal of these endpoints is to enable the independent AI features in self-managed instances within the proposed timeline. See the issue and this issue for more information.

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Screenshots or screen recordings

Test Anthropic::Client from the Rails console:

[3] pry(main)> Gitlab::Llm::Anthropic::Client.new(User.first, unit_primitive: 'generate_description').stream(prompt: "\n\nHuman: Hi, How are you?\n\nAssistant:")
=> " I'm doing well, thanks for asking!"
[2] pry(main)> Gitlab::Llm::Anthropic::Client.new(User.first, unit_primitive: 'generate_description').complete(prompt: "\n\nHuman: Hi, How are you?\n\nAssistant:")
=> {"type"=>"completion",
 "id"=>"compl_01K4w7PbhRsNPWwgQ5BWUhsx",
 "completion"=>" I'm doing well, thanks for asking!",
 "stop_reason"=>"stop_sequence",
 "model"=>"claude-2.1",
 "stop"=>"\n\nHuman:",
 "log_id"=>"compl_01K4w7PbhRsNPWwgQ5BWUhsx"}
AI Gateway log

Access request log:

[3] pry(main)> Gitlab::Llm::Anthropic::Client.new(User.first, unit_primitive: 'generate_description').stream(prompt: "\n\nHuman: Hi, How are you?\n\nAssistant:")
=> " I'm doing well, thanks for asking!"

{
    "url": "http://localhost:5052/v1/proxy/anthropic/v1/complete",
    "path": "/v1/proxy/anthropic/v1/complete",
    "status_code": 200,
    "method": "POST",
    "correlation_id": "612c690bc81254ce75bbd98d7736656f",
    "http_version": "1.1",
    "client_ip": "127.0.0.1",
    "client_port": 57456,
    "duration_s": 0.5752928909969341,
    "duration_request": -1,
    "cpu_s": 0.011907270000000025,
    "user_agent": "Ruby",
    "gitlab_instance_id": "ba75b213-4fd4-4311-8631-0ac7a1bd3247",
    "gitlab_global_user_id": "Cv4L37An7TsFzTjzy4yCixBZwUsK8+TCQYl7EYHVN8c=",
    "gitlab_host_name": "gdk.test",
    "gitlab_version": "17.1.0",
    "gitlab_saas_duo_pro_namespace_ids": null,
    "gitlab_saas_namespace_ids": null,
    "gitlab_realm": "saas",
    "meta.feature_category": "ai_abstraction_layer",
    "meta.unit_primitive": "generate_description",
    "logger": "api.access",
    "level": "info",
    "type": "mlops",
    "stage": "main",
    "timestamp": "2024-05-20T05:45:45.145145Z",
    "message": "127.0.0.1:57456 - \"POST /v1/proxy/anthropic/v1/complete HTTP/1.1\" 200"
}

Proxy request log:

{
    "correlation_id": "612c690bc81254ce75bbd98d7736656f",
    "logger": "httpx",
    "level": "info",
    "type": "mlops",
    "stage": "main",
    "timestamp": "2024-05-20T05:45:45.143625Z",
    "message": "HTTP Request: POST https://api.anthropic.com/v1/complete \"HTTP/1.1 200 OK\""
}

How to set up and validate locally

  1. Checkout feat: use OR expression for the required scope ... (gitlab-org/modelops/applied-ml/code-suggestions/ai-assist!814 - merged) in AI Gateway.
  2. Checkout this MR in GitLab-Rails.
  3. Enable the feature flag ::Feature.enable(:use_ai_gateway_proxy).
  4. Follow Optional: Test with OIDC authentication section.

Further reading

Edited by Michael Thomas

Merge request reports