Skip to content

feat: add code generation client to test xray RAG with libraries

Missy Davies requested to merge md-add-code-gen-client into main

What does this merge request do and why?

Resolves #19

Add a second anthropic client with libraries hard coded in as further prompt enhancements in order to explicitely test x-ray RAG and facilitate fine tuning prompts.

The prompt is based off of the current prompt in the monolith: https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/code_suggestions/prompts/code_generation/anthropic_messages.rb#L36

How to set up and validate locally

  1. Run anthropic_code_gen evaluation
poetry run eli5 code-suggestions evaluate \
  --dataset="missy-test-rag" \
  --source=anthropic_code_gen \
  --limit=2 \
  --offset=0 \
  --experiment-prefix=mcdd \
  --intent="generation" \
  --rate-limit=29
  1. Run standard anthropic evaluation
poetry run eli5 code-suggestions evaluate \
  --dataset="missy-test-rag" \
  --source=anthropic \
  --limit=2 \
  --offset=0 \
  --experiment-prefix=mcdd \
  --intent="generation" \
  --rate-limit=29
  1. Compare results in LangSmith

Screenshot_2024-07-11_at_11.21.22_AM

Screenshot_2024-07-11_at_11.13.50_AM

For example, note the client that includes enhanced context with libraries (right) correctly identifies and uses the GraphQl gem in the output: Screenshot_2024-07-11_at_11.15.27_AM

The input question was: "# I have a database table named 'projects' with columns 'id', 'name', 'description', and 'sign_in_count'. Write an API resource type that exposes the fields of this table."

Merge request checklist

  • Tests added for new functionality. If not, please raise an issue to follow up.
  • Documentation added/updated, if needed.
Edited by Missy Davies

Merge request reports