Skip to content

Move chat tests from rake task to rspec

euko requested to merge 419660-zeroshot-tool-picking-spec into master

What does this MR do and why?

  • Migrates the questions ee/lib/tasks/gitlab/llm/questions.yml previously tested by a Rake test over to the zeroshot executor Rspec.

  • Adds extra questions and string match tests

    • Ex. 'Can you explain the code ""def hello_world\nputs(""Hello, world!\n"");\nend""?' | [] | /ruby/i
  • Adds the Rake test to populate tanuki_bot_mvc table where embeddings are stored from a pre-generated set of embeddings.

  • Adds another Rake test that generates a test fixture that contains the embeddings only relevant for the test questions.

    • See the updated documentation for details.

How to set up and validate locally

Run the rspec:

export OPENAP_API_KEY='<key>'
export ANTHROPIC_API_KEY='<key>'
REAL_AI_REQUEST=1 rspec ee/spec/lib/gitlab/llm/chain/agents/zero_shot/executor_spec.rb

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #419660

Edited by euko

Merge request reports