New Data Set: Questions on MRs

As the Chat team expands context for what Duo Chat can use to answer questions, there are multiple areas opening up for testing. One area is for questions on MRs. In order to validate performance on this metric, Chat needs a dataset that represents questions that one might ask about an MR.

Proposal

• ✅ MVC: we will simply use the questions from the chat bashes and in product feedback as a very, very basic test set to sanity check our solution
• We will focus on historical data to create a dataset, leveraging the GitLab MR API.
• We will pull out and parse comments on MRs, and filter the data to just questions or rephrase comments as questions.
• We will use metrics similar to that of previous use cases for Chat such as Docs or Issue/Epics
• We will add the dataset to the Chat pipeline and dashboard.

Questions from Duo Chat bashes and in-product feedback

Data from chat bashes

The following is a list of MR-related questions from all chat bashes that do only require the local context of one MR not the global context of all MRs:

No.	Date of Bash	Query	Chat response	How could Duo Chat's response been improved?	Overall Quality	Deep Link to the query
2	7/31/2023	Can you summarize this MR? gitlab-org/gitlab!127795 (merged)	I am sorry, I am unable to find the issue you are looking for.	I'd expect Duo Chat to be able to find an MR in a public project like this.	2	J18
3	7/31/2023	Can you summarize the progress of gitlab-org/gitlab!127220 (merged) and what is left to do? (from within the actual MR detail page)	I am sorry, I am unable to find the issue you are looking for.	Is this bot only aware of issues?	2	J31
4	8/8/2023	Is this MR merged ? (obviously not a typical user question but a tester's question to see if chat works; so no need to optimize for this)	I am sorry, I am unable to find the issue you are looking for.	I think this is something that I just checked since the pipelines were running, I did not expect an answer but would be surprised if it did	3	J13
5	8/8/2023	Can you assign this MR to a reviewer ?	Here are the steps to assign a reviewer to a merge request in GitLab: Navigate to the merge request you want to assign a reviewer to. On the right side of the MR, click on the 'Reviewers' dropdown. Start typing the name or username of the person you want to add as a reviewer. Select their name when it appears. Click the 'Assign' button to assign the reviewer. The reviewer will be notified and asked to review the changes in the MR. So in summary, from the merge request, use the 'Reviewers' dropdown to search for and select the reviewer, then click 'Assign' to assign them.	It is good, although I would love it to assign to the suggested reviewer as well :D Too much to ask ?	4	J16
6	8/24/2023	what's wrong with this MR?	I am sorry, I am unable to find the issue you are looking for.	It has no awareness of what page I am on or merge requests in general	1	K9
7	8/24/2023	can you summarize this merge request? https://gitlab.com/gitlab-org/analytics-section/product-analytics/devkit/-/merge_requests/74	I don't see how I can help. Please give better instructions!	There was already a summary posted to the MR, yet Duo Chat couldn't find it. Also, it couldn't discern which MR I was looking at without me copy and pasting the URL	1	K14
8	8/24/2023	is there a failure with this merge request? gitlab-org/gitlab!129981 (merged)	I don't see how I can help. Please give better instructions!	It could check the content of the merge request to see if it detects an error	1	K30
9	9/12/2023	What is the link to the lastest output of the pipeline build of this MR	I am sorry, I am unable to find the issue you are looking for.	A link to something or a way to find what I asked about.	2	K12
10	9/12/2023	Who made this MR? (obviously not a typical user question but a tester's question to see if chat works; so no need to optimize for this)	I am sorry, I am unable to find the issue you are looking for.	Anwer my question	2	K13
11	2/28/2024	Summarize what the MR 4764 is trying to accomplish	The question you are asking requires data that is not available to GitLab Duo Chat. Please share your feedback below.	Duo told me in a previous response to give it an MR id and it would help me summarize it. I give it the id and then it told me it can't help. Duo should be clear from the start that it doesn't have access to specific pages.	4	S14
12	3/26/2024	Can you suggest reviewers for this MR?	Unfortunately, I do not have enough context to suggest appropriate reviewers for a merge request without more details. Some additional information that would be helpful includes: The ID or URL of the merge request in question The project or repository the MR is associated with The files, code changes, or other content included in the MR Any teams, developers, or subject matter experts that may be relevant reviewers based on the content Without access to the actual merge request or surrounding details, I cannot make informed suggestions for reviewers. Please	I thought suggesting reviewers was one of Duo Chat's features. Not sure why this doesn't work. Maybe make it clear how to make it work in your documentation.	2	F57
13	3/26/2024	Can you identify vulnerability of this MR: https://gitlab.com/virginmediaO2/advanced-analytics/dbt-repositories/network-intelligence-analytics/network-data-products/-/merge_requests/1237	Unfortunately, I do not currently have access to vulnerabilities or merge requests in GitLab. As an AI assistant named "GitLab Duo Chat", I can only access issues, documentation, epics, and CI editor resources. Without being able to view the actual merge request, I cannot identify any vulnerabilities it may contain. I apologize that I am unable to assist with answering this specific question. Please feel free to ask another question that I can help with using my permitted GitLab access.	Again, I thought identifying vulnerability was one of the Duo Chat's features. Maybe provide more instructions in the chat to teach the users how this may work?	2	F58
14	4/17/2024	can you describe the MR for me?	Unfortunately, I do not have access to view merge requests in GitLab. So I cannot provide any details to describe a specific MR without more context. I apologize that my capabilities are limited when it comes to accessing certain GitLab resources like merge requests and pipelines. Please feel free to ask any other questions related to issues, documentation, epics or CI configuration that I may be able to assist with using the tools available to me. If there is any other way I can help, don't hesitate to ask!	there were at some point experimental Duo feature that tried to explain MRs or lines of code, i thought it's integrated here as well..	1	G26
15	4/17/2024	What is this MR about?	Unfortunately, I do not have access to information about merge requests in my current limited form. As the GitLab Duo Chat assistant created by GitLab, I can provide details on issues, documentation, epics, and CI configuration. However, without more context or an issue ID related to this MR, I cannot determine what it is about. Please feel free to provide an issue ID or more details that could help me understand the context of this MR. I would be happy to assist further once I have additional information to work with. If I still cannot address your question, I apologize for the limitation. Please leave feedback so I can continue to improve.	Understanding the context of the page I was on and integrating with the MR summary tool.	2	G45
16	4/17/2024	can you describe this MR and what it is doing?	Unfortunately, I do not currently have access to any merge requests or their details. As an AI assistant focused on DevSecOps and created by GitLab, I can provide helpful information related to issues, documentation, epics, CI pipelines, and other software development topics. However, without access to the specific merge request you asked about, I cannot describe it or what it is doing. I apologize that I could not directly answer your question. If there is any other GitLab-related information you need, please feel free to ask. And if possible, please leave feedback to help improve my capabilities - perhaps enabling future access to merge requests could allow me to assist with similar questions.	If there is functionality for summarization/explanation of an MR, point me to how to do that from the response. I EXPECT to have the AI be aware of what I am doing, and for it to say "due to privacy I can't help you" is not a good look.	2	G611

Questions mentioned in the in-product feedback

The following is a short list of MR-related questions that are not already covered in the above list

What were you doing?	How could the response be improved?	comment
Asking for code review	Take the MR I was on and give actual code feedback
Asking to explain code	It refused to answer
Asking about comments in open mr	Make it work. this is absolutely useless
What is the purpose of this MR/ticket	I was on an MR with the ticket in the name of the MR. There is a link to the ticket on the webpage I am on. The chatbot should have access to the ticket the MR is about. It should answer the question	This question requires more context than the local MR context. Chat would need to also load the context from the referenced issue.