Standardize a UX pattern to give feedback
Context
As part of the AI features, we quickly developed a way for the user to give feedback on the AI's response. This issue will encompass the design work to standardize this into a repeatable experience (if possible).
Problem to solve
AI can generate good and bad responses.
We want to understand the quality of the responses to understand where we need to make improvements. We also want to know if AI is generating content that is offensive/abusive or is otherwise a legal issue.
Click to read about other feedback avenues
A feedback mechanism in the UI is not the only way we will evaluate the quality of the responses. We are not getting much engagement of the feedback mechanism. We can also use:
- user sessions
- public feedback issues
- usage analytics
User testing
Click to read about user testing
User testing week of 2023-06-19
I ran moderated testing of 2 feedback mechanisms with 6 GitLab staff.
Findings:
- Participants were more motivated to share feedback when they believed their feedback would make the chat better, for example by training the model.
- Participants were uncertain of where feedback goes.
- Participants may not be motivated to give feedback if the response from the AI was not entirely incorrect, but maybe not helpful. When asked if a participant would give feedback on an incomplete answer, they said “No, because like, it's not giving me a completely wrong answer.”
- Participants want to share context with their feedback
- Participants found it difficult to categorize their feedback simply into helpful / unhelpful / wrong (when testing the
👍 👎 version)
User testing week of 2023-07-03
Following the moderated user testing, I ran unmoderated user testing on the following prototype with external users.
details of designs tested
| Design | Finding |
|---|---|
 |
User is motivated to give feedback if they believe it will help |
 |
User is motivated to give feedback if they believe it will help |
 |
User wants to know what happens to feedback |
 |
User wants to provide context |
Findings
- Giving feedback was discoverable
- participants did not opt to give feedback often in the test. However, participants can imagine themselves giving feedback. The most commonly cited reason for giving feedback was if an answer was factually incorrect. Insight.
- Participants expected their feedback to be reviewed by GitLab staff. They expected their feedback to improve the quality of the answers they received. Insight.
Design proposal
- Each AI feature has a call to action under the content
- The CTA links to a modal
- When the modal is submitted, the user sees a confirmation where the original CTA was
- When the modal is submitted, a toast is shown
- The modal should be the same across features, unless there is a good reason for it to be different
| Feature | CTA | Confirmation |
|---|---|---|
| chat |  |
 |
| explain vulnerability |  |
 |
| summarize MR |  |
 |
| summarize comments |  |
 |
Considerations
Why not link to a feedback issue?
Feedback issues are an easy way to collect feedback. However, they can suffer from confirmation bias and group-think. It's better to collect user feedback in isolation, where the user is not being influenced by other's thoughts and opinions.