Move chat tests from rake task to rspec
What does this MR do and why?
-
Migrates the questions
ee/lib/tasks/gitlab/llm/questions.yml
previously tested by a Rake test over to the zeroshot executor Rspec.- Omits the test cases that are failing. See: https://gitlab.com/gitlab-org/gitlab/-/issues/419660#note_1493359919
-
Adds extra questions and string match tests
- Ex. 'Can you explain the code ""def hello_world\nputs(""Hello, world!\n"");\nend""?' | [] | /ruby/i
-
Adds the Rake test to populate
tanuki_bot_mvc
table where embeddings are stored from a pre-generated set of embeddings. -
Adds another Rake test that generates a test fixture that contains the embeddings only relevant for the test questions.
- See the updated documentation for details.
How to set up and validate locally
Run the rspec:
export OPENAP_API_KEY='<key>'
export ANTHROPIC_API_KEY='<key>'
REAL_AI_REQUEST=1 rspec ee/spec/lib/gitlab/llm/chain/agents/zero_shot/executor_spec.rb
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.
Related to #419660
Edited by euko