Skip to content

Add extra questions to test Chat's selection of the documentation tool

Bruno Cardoso requested to merge bc/duo-chat-add-doc-questions into master

What does this MR do and why?

Ref #434021 (closed)

Add extra questions to test the selection of the GitlabDocumentation tool. Questions were sourced from this sheet: https://docs.google.com/spreadsheets/d/1jm059l3T9XL2aj5VgBsqVgcumbiqYDZ0c-Utyrg0u-o/edit#gid=1099891908.

For later reference, this is the current result for the questions:

ANTHROPIC_API_KEY="${ANTHROPIC_TOKEN}"  \
VERTEX_AI_PROJECT='ai-enablement-dev-69497ba7' \
LLM_DEBUG=1 \
REAL_AI_REQUEST=1 \
bin/rspec ee/spec/lib/gitlab/llm/completions/chat_real_requests_spec.rb:278

...

Finished in 12 minutes 56 seconds (files took 10.38 seconds to load)
40 examples, 1 failures

Failed examples:

rspec './ee/spec/lib/gitlab/llm/completions/chat_real_requests_spec.rb[1:1:5:5:1:1]' # Gitlab::Llm::Completions::Chat real requests when asking about how to use GitLab input_template: "What is DevOps? What is DevSecOps?", tools: ["GitlabDocumentation"] behaves like successful prompt processing answers query using expected tools

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

How to set up and validate locally

  1. Run the test just for the documentation questions.
ANTHROPIC_API_KEY="${ANTHROPIC_TOKEN}"  \
VERTEX_AI_PROJECT='ai-enablement-dev-69497ba7' \
LLM_DEBUG=1 \
REAL_AI_REQUEST=1 \
bin/rspec ee/spec/lib/gitlab/llm/completions/chat_real_requests_spec.rb:278
Edited by Bruno Cardoso

Merge request reports