Investigation of Tool Selection Success Rate
Objective:
The tool selection spec success rate decreased after the merge of gitlab-org/gitlab!145959 (comment 1794603728).
The objective is to see if we can improve tool selection without sacrificing overall metrics.
Metric:
Less tool selection failures when running ee/spec/lib/gitlab/llm/completions/chat_real_requests_spec.rb
Dataset:
Metrics:
1.Control Metric Score: On master we see 7~9 failures. (Note that before the change we see 5 failures.)
2.Experiment Metric Score: TBD post experiment
3.Variance: TBD post experiment
Experiment Details:
Recommendation: Revise the change in gitlab-org/gitlab!145959 (merged) and see if any change caused the success rate, and see if we can preserve fragments of the changed prompt.