Skip to content

Modify plan tools to use Claude 3

Nathan Weinshenker requested to merge jy-issue-epic-tool-conversion into master

What does this MR do and why?

This MR aims at migrating all the existing Plan tooling to utilize the Claude 3 Haiku model. I'm not exactly sure what the basic for using the Haiku model, but evaluation testing has shown it's more performant with these tools.

The goal is a wider push to make sure we are migrating to Claude 3 as a collective org.

#455112 (closed)

MR acceptance checklist

  • Ensure the following tools list in the MR work locally
  • Make sure that the response format is coming in useful format.

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

Tool Before FF After FF Lang Trace
Issue Reader Screenshot_2024-04-22_at_4.45.43_PM Screenshot_2024-04-22_at_4.52.32_PM https://smith.langchain.com/o/0526640f-6cc5-589e-b08c-3cf00d6d99cf/projects/p/7af1f53b-8019-44cb-8293-a174b705809a?timeModel=%7B%22duration%22%3A%227d%22%7D&peek=3e659fd0-6a97-4849-bfe7-29ec30138197
Epic Reader Screenshot_2024-04-22_at_4.56.34_PM Screenshot_2024-04-22_at_4.56.34_PM https://smith.langchain.com/o/0526640f-6cc5-589e-b08c-3cf00d6d99cf/projects/p/7af1f53b-8019-44cb-8293-a174b705809a?timeModel=%7B%22duration%22%3A%227d%22%7D&peek=0b0d7905-d0c0-4a16-ab0f-83a71135e22c

How to set up and validate locally

  1. In rails console enable the experiment fully
    Feature.enable(:ai_claude_3_sonnet)
  2. Visit the following group that has Duo Enabled features such as http://127.0.0.1:3000/test
  3. Ensure that following prompts are coming from Claude 3 generated results are coming with the following format
[
  {"role": "user", "content": "Hello there."},
  {"role": "assistant", "content": "Hi, I'm Claude. How can I help you?"},
  {"role": "user", "content": "Can you explain LLMs in plain English?"},
]
Edited by Jessie Young

Merge request reports