Add a real-model arm so the AI gateway can call a real provider

Andrew Dunnrequested to merge
feature/real-model-arm into main

The mock gateway answers questions about wiring and nothing else. Latency, refusals, token limits and the shape a provider error has by the time it reaches the UI all need a real provider behind the gateway. This adds an arm that swaps one for the other and swaps back, on either of two routes, with no long-lived secret on the VM.

just real-model vertex            # instance service account, then one completion to prove it
just real-model proxy vertex      # LiteLLM on loopback between gateway and provider
just real-model param-strip       # one request twice, with and without temperature
just real-model off               # back to the mock

Vertex authenticates as the VM through Application Default Credentials; Bedrock assumes a federated role with a GCE identity token that a systemd timer keeps current. Direct mode is the customer path and where a finding lands. Proxy mode is the isolating instrument: the composed request otherwise exists only inside the gateway container, so a refusal reads the same whether the credential is wrong or the model has stopped accepting a parameter.

flowchart LR
    R["GitLab Rails"] --> G["AI gateway"]
    G -- "direct: the customer path" --> P["provider<br/>Vertex or Bedrock"]
    G -- "proxy: the instrument" --> L["LiteLLM on loopback"]
    L --> P
    L -. "request log makes the<br/>composed request readable" .-> O["attribution:<br/>proxy or provider"]

What the lab showed

Two caproni runs, every arm exercised, both instances torn down and verified.

arm verdict
mock control passes, 0.41 s
Vertex direct, claude-sonnet-4-6@default passes, 1.64 s, location resolves to global
Vertex direct on us-east5 passes
proxy mode passes, and the seam assert fires: the proxy logged the request
param-strip on claude-opus-4-7@default reproduces, refusal attributed to the proxy
doctor fail arm, a model card that was never enabled refuses, exit 1
just doctor 46 pass, 0 fail, no upstream drift

param-strip probes gitlab-org/gitlab#604985, temperature on Opus 4.7 and newer. Two things the runs settled about that claim. The refusal the arm sees comes from LiteLLM, client-side, before the provider is reached, so the verdict names the layer rather than asserting the provider refused. Called directly, the provider does refuse: claude-opus-4-7 answers 400 temperature is deprecated for this model for every value except 1, and so do claude-opus-4-8 and claude-sonnet-5, while claude-sonnet-4-6 accepts it. The deprecation is wider than the issue records.

Four defects the lab found in the arm itself, fixed here

  • param-strip claimed the provider refused a parameter when the refusal was the proxy's. It now attributes the layer, and an unattributable refusal is inconclusive rather than a pass.
  • The proxy's key-safety assert demanded a 401 the pinned LiteLLM never sends: with no database behind it, a non-master key is turned away as 400 no_db_connection. The assert accepts that error type and no other 400, and still aborts on a 200.
  • openssl is absent from the base image, so minting the proxy key produced the literal sk-, which passed a prefix-only guard. The key now mints from /dev/urandom and the guard asserts a length.
  • The arm needed a Cloud Connector key that only the agent-platform chain created, so its first probe failed on a stock instance. The arm ensures one.

Also here: the Vertex location default now promotes on any Anthropic id in the set rather than the primary alone, since the proxy's deployments share one location and a Gemini primary with a Claude strip model resolved somewhere Claude does not answer.

Known limits

  • The Bedrock route is code-complete and unvalidated end to end. Its federation, token refresher and recipes are here and the role exchange is proven, but the sandbox account available for this work is denied the Anthropic first-time-use form by an organization service control policy, so no model agreement can exist in it. The arm reports that as a named inconclusive rather than a pass. Validating it needs an account where Bedrock model access already exists.
  • The key-safety assert's 401 branch has never fired, and is carried for a LiteLLM version that grows one. Recorded in the maintenance register.
  • The add-on seat disagreement the runs hit is #8, filed separately.

Testing

just ci green on every commit. Lab evidence above from two provisioned instances; the shell gates now cover the proxy subdirectory, which they did not.

Merge request reports