Skip to content

Modify ETV to use anthropic body streaming to fix timeout problems

Gregory Havenga requested to merge ghavenga-anthropic-body-streaming into master

What does this MR do and why?

During limited rollout of the Anthropic client for Explain This Vulnerability for testing purposes, it was discovered that it seemed to not work at all in Staging or Production. This was later established to have been caused by Net::ReadTimeout errors being raised due to the api taking too long to service the request.

This MR uses the new body streaming mechanics established by the AI framework team to collect the full response before processing, avoiding the timeout issue, and also opening the possibility to UI response streaming if desired.

If we do somehow still have timeout issues, a more specific failure message is presented in the response to simplify diagnosis.

How to set up and validate locally

Assuming a local environment with SAST vulnerabilities and the ability to execute Anthropic client requests, your requests should still work, but should now no longer timeout (though they may take a very very long time.)

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Relates to https://gitlab.com/gitlab-org/gitlab/-/issues/419597

Edited by Gregory Havenga

Merge request reports