feat: add code generation client to test xray RAG with libraries
What does this merge request do and why?
Resolves #19 (closed)
Add a second anthropic client with libraries hard coded in as further prompt enhancements in order to explicitely test x-ray RAG and facilitate fine tuning prompts.
The prompt is based off of the current prompt in the monolith: https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/code_suggestions/prompts/code_generation/anthropic_messages.rb#L36
How to set up and validate locally
- Run
anthropic_code_gen
evaluation
poetry run eli5 code-suggestions evaluate \
--dataset="missy-test-rag" \
--source=anthropic_code_gen \
--limit=3 \
--offset=0 \
--experiment-prefix=yes-rag \
--intent="generation" \
--rate-limit=29
- Run standard
anthropic
evaluation
poetry run eli5 code-suggestions evaluate \
--dataset="missy-test-rag" \
--source=anthropic \
--limit=3 \
--offset=0 \
--experiment-prefix=no-rag \
--intent="generation" \
--rate-limit=29
- Compare results in LangSmith
For example, note the client that includes enhanced context with libraries (right) correctly identifies and uses the GraphQl gem in the output:
The input question was: "# I have a database table named 'projects' with columns 'id', 'name', 'description', and 'sign_in_count'. Write an API resource type that exposes the fields of this table."
Merge request checklist
-
Tests added for new functionality. If not, please raise an issue to follow up. -
Documentation added/updated, if needed.
Edited by Missy Davies