Update prompt to include the "why" of the MR
Background
We want to add more context to the prompt for review so model can have more idea on what the changes are for.
Example feedback
Feedback: #517386 (comment 2370365831)
Duo doesn't read the merge request title? It doesn't understand the feature flag was being removed mention in the title. !181148 (comment 2370363413)
Feedback: #493890 (comment 2136451449)
I tried this feature on !167248 (merged)
The main feedback was that the change I made might have security implications. Which is maybe a good warning! But I wonder if, had the bot been able to view the related issue, it would have known that this change was intentional.
Feedback: #463039 (comment 2124044806)
Tried on my small golang MR: cli!1693 (merged). 2 comments were posted:
1 priority
7: not useful, caused by the LLM not having context of the goal of the MR/issue. cli!1693 (comment 2124029998)
Feedback: #493890 (comment 2244224081)
For this MR, the intent is to remove code and all the comments from duo are about the code that's intentionally removed. I'll admit most of the context is in the issue that's linked so it might not be that straightforward to know?
Proposal
Experiment in Langsmith with adding various combinations of the following pieces of context:
- MR title
- MR description
- MR description summary generated by LLM? - https://docs.gitlab.com/ee/user/gitlab_duo/index.html#merge-request-summary
- Issue title
- Issue description
- Issue description summary generated by LLM?
Experiment with using Duo Chat's MergeRequestReader tool to provide this context, see #510161 (comment 2324589687).
Consider using the following sample MRs as dataset to manually evaluate whether the extra context improves the comments:
Also, whatever changes we make here needs to included in the doc