Skip to content

Address Duo Chat issue/epic searching problem

Lesley Razzaghian requested to merge 457756-duochat-emptying-assistant-answer-unpredictably-when-answer-is-streaming-in into master

What does this MR do and why?

This MR includes prompt changes to solve the problem where Duo Chat was attempting to search epics or issues, even though it does not have that capability.

This can cause numerous errors.

BEFORE

imageimage

AFTER

imageimage

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Screenshots or screen recordings

Here are screenshots of the Python Notebook evaluations I did for these prompt changes.

Here is the Python Notebook I used.

Before

Issue questions - before

image

Success count: 30/30 ✅

Epic questions - before

Screenshot_2024-05-25_at_1.17.13_AM

Success count: 30/30 ✅

Avoiding attempting to search - before

image

Success count: 0/30 ❌

After

Issue questions - after

image

Success count: 30/30 ✅

Epic questions - after

image

Success count: 30/30 ✅

Avoiding attempting to search - after

image

Success count: 30/30 ✅✅✅

How to set up and validate locally

  1. Enable prevent_issue_epic_search feature flag

  2. Note that questions such as find me the gitlab-org/gitlab issue that is about the API not being able to tell us which subscription the user has do not error out

Related to #457756 (closed)

Edited by Lesley Razzaghian

Merge request reports