Problem/Solution validation slash commands Duo Chat

Findings

• Slides
• Actionable insights

What’s this issue all about?

We are working on a standard pattern of slash commands (or quick actions) in GitLab Duo Chat but we do not fully understand the use cases for these quick actions. This issue proposes a hybrid problem and solution validation to quickly gather more data on the user needs and usability test slash commands.

What are slash commands?

Who is the target user of the feature?

• Users of GitLab Duo Chat (ultimate tier, have GitLab Duo Pro seat)
• Primarily the software engineer persona
• Use Duo Chat in either the IDE or GitLab

What questions are you trying to answer?

Problem validation

• What are the primary use cases for slash commands?
  • When would a user use a slash command versus typing in natural language?
  • What are the benefits of slash commands?
• In what cases does the user want to attach an object (e.g. a link) to a slash command versus how common is it to ask about the object they are currently looking at?
• Does the user want to add additional context to their slash command? (e.g. "summarize this issue in 5 bullet points")

Solution validation

• Do users discover the slash command menu
• Do users understand the available slash commands
• Do users understand how to add an object to a slash command (e.g. link, additional information, issue ID...)

Core questions

• What are the primary use cases for slash commands?
• What are the main pain points with the existing UX paradigms of adding objects to slash commands

What hypotheses and/or assumptions do you have?

• Users will primarily use slash commands on the object they are currently looking at and will not need to attach links or other objects to their slash commands
• Slash commands are primarily and efficiency tool and we should focus on slash commands working without arguments

What decisions will you make based on the research findings?

• Understand which slash commands should be available by default
• Understand how much to prioritize improving the existing UX to attach links/objects to slash commands

What's the latest milestone that the research will still be useful to you?

Discussion guide and screener

• Discussion guide
• Screener
• Recruitment issue
• Prototype

changed milestone to %17.2

added AI-UX groupduo chat labels

set weight to 2

assigned to @katiemacoy

added Category:Duo Chat devopsai-powered sectiondata-science labels

changed the description

changed title from DRAFT Problem/Solution validation slash commands Duo Chat to Problem/Solution validation slash commands Duo Chat

changed the description

@NickHertz I'm curious to get your feedback on a study I want to run. I am trying to figure out the default slash commands available in Duo Chat. I'm thinking about running a mixed problem and solution validation study.

Firstly to understand the use cases for slash commands in general. Some of the open questions are how common is it to add "arguments" to slash commands:

I would also like to test the usability of slash commands.

I'd love your thoughts on my approach and discussion guide.

cc @nickleonard @allison.browne if you have any feedback - I'm going to try to build in some questions about what users want summarized on an issue too.

I'm going to try to build in some questions about what users want summarized on an issue too.

@katiemacoy That sounds great. I'd be curious to learn as well how users want summaries, e.g. in your example "in 5 bullet points" relates to the format of the output rather than the content to summarize. That might not fit here, but something we can keep an eye out for.

In terms of what to summarize, I have a hypothesis that time is relevant — updates from the last week, or updates since I last saw this — which would be interesting to see if we learn anything that would confirm this. Realize that isn't the focus of the study, but if we happen to hear this!

cc @gweaver in case you have any other thoughts here

@karenyli this is a study I would love to run soon with the AI research panel

Nick - I tagged you on the discussion guide - please feel free to add your additional questions directly into the script. We can do a final edit of the script when everyone has added their input!

In terms of what to summarize, I have a hypothesis that time is relevant — updates from the last week, or updates since I last saw this — which would be interesting to see if we learn anything that would confirm this.

@katiemacoy @nickleonard I hypothesize that people will be interested in having specific insights they want to be answered with a summary, such as:

• Summarize "whats new" since i last view the object.
• Summarize the current status and any blockers that need attention.
• Summarize what the object is about (description).
• Summarize the decisions relating to specific topics within the object.
• Summarize outstanding tasks or action items that have been completed, or are waiting to be completed.
• Summarize open questions that haven't been addressed.

My hypothesis is largely based on what I've anecdotally learned from speaking with customers and observing organic solutions such as https://gitlab.com/gitlab-org/secure/pocs/discussion-automation, which are aimed at summarizing such things as topics and resolutions on each topic thread.

@katiemacoy Awesome. I've just created the issue #3102 to get you started. :)

@katiemacoy your discussion guide looks great! I think it makes total sense to combine the problem validation and the solution validation for this topic. I think your tasks are well written and I am excited to see what people's responses are.

mentioned in issue gitlab#467389 (closed)

changed the description

mentioned in issue gitlab#468439 (closed)

mentioned in issue gitlab#469236 (closed)

marked this issue as related to gitlab#462122 (closed)

mentioned in issue gitlab#462122 (closed)

added UX research label

added uxr-progressrecruiting/fielding label

changed the description

mentioned in merge request gitlab-org/gitlab-services/design.gitlab.com!3909 (merged)

marked this issue as related to gitlab#470705

mentioned in issue gitlab#470705

mentioned in issue gitlab#469050 (closed)

mentioned in issue gitlab-design#2591 (closed)

added uxr-progressanalysing label and removed uxr-progressrecruiting/fielding label

changed the description

changed the description

Actionable insights & related epics/issues

(Full details: Google slides)

Priority	Actionable insight	Recommendation
High	Participants were not aware of the in-product chat's capabilities and had not adopted the chat	Drive adoption by building more shortcuts into the UI to meet users where they are in their normal workflow, perhaps start with a single call-to-action and measure the impact
High	Participants explored the chat through trial-and-error behavior, which could lead to frustration	Drive "happy path" adoption of the chat, such via as shortcuts or prompts. Set expectations of capability before the user asks a question
Med	Participants were uncertain if the chat had context of their current page	Inform users what context the chat has or does not have before they ask their question
Med	Participants work with a broad context and expected or hoped Duo would do the same	Identify key workflows to understand user needs for context awareness
Med	Confusion about the meaning and difference between slash commands	Inform users of any changes to existing slash commands, support multiple commands in certain workflows
Low	It’s important to be able to cross-reference the AI’s output with the content it’s referring to	Consider making it easier to cross-reference content by adding citations
Low	Improve issue summary content	Improve content to better address user's needs
Low	Improve issue summary format	Shorten content
Low	Slash commands are not discoverable	Improve discoverability of slash commands
N/a	Adding additional information to a slash command is not discoverable	Possibly no action, participants who wanted to ask specific questions seemed to prefer to type in natural language
N/a	Participants were confused by slash commands that did not pertain to their current page	No action, product decision already made was to only show commands related to current page on the slash command menu

@tlinz Can you keep this comment on your radar as you put together plans for the next few quarters? These findings give us some good starting places to improve chat (some of them are very specific to slash commands and others are more general.

It's also worth looking at the slides again (15-27 are about slash commands). In general, slash commands didn't perform so well in research that I'd say they are a solution we need to double down on. We could consider something different altogether.

cc @rogerwoo @susie.bee

changed the description

added uxr-progresspublished label and removed uxr-progressanalysing label

closed

unassigned @katiemacoy

removed milestone %17.2