Add MergeRequestReader to chat
Compare changes
Files
2- Gosia Ksionek authored
@@ -8,13 +8,13 @@ module MergeRequestReader
Addresses one popular context the chat currently does not support: https://gitlab.com/gitlab-org/ai-powered/duo-chat/discussions/-/issues/3+
Specifically this MR addresses Support Merge Requests as context for Duo Chat ... (#464587 - closed) • Lesley Razzaghian • 17.5 • Needs attention
Here are the evaluation results from a collective LLM judge on the master branch
Here are the results on this branch
Here are the stats (averages)
Master branch
Correctness: 3.65
Readability: 3.75
Comprehensiveness: 3.43
This branch
Correctness: 3.71
Readability: 3.75
Comprehensiveness: 3.53
The improved results in the existing evaluation are likely not statistically significant, but at least it proves these changes do not degrade existing questions.
Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
Screenshots are required for UI changes, and strongly recommended for all other merge requests.
Before | After |
---|---|
Numbered steps to set up and validate the change are strongly suggested.
Feature.enable(:ai_merge_request_reader_for_chat)
Task | Status | Notes |
---|---|---|
Run CEF locally on this branch vs master to ensure no degradation | ![]() |
See above in 'Evaluation results' |
Create basic dataset for MR eval | ![]() |
Here |
Change Chat REST API to accept MR requests | ![]() |
|
Add seed data for merge requests to be able to test locally in CEF | ![]() |
I believe this can be a followup |
Make this work with 'v2_chat_agent_integration' feature flag on | ![]() |
This MR just needs to be merged after this one has been rolled out |
Ask model validation team to add this dataset to daily runs, merge this MR and monitor eval results | ![]() |
Asked them to add it discussion |