Internal Developer Testing for Duo Context Feature

Learnings (from 3 testers)

User testing was conducted with three developers. The primary use cases tested were debugging import errors and generating test cases.

🎯 What's Working Well

  1. Quality of AI Responses
    • Generated high-quality test cases for Vue components
    • Provided detailed debugging assistance
    • Developers found the outputs accurate and valuable
  2. File Selection
    • Users could successfully select individual files
    • The UI gives a clear indication that files were included as context

🚨 Pain Points

  1. File Selection UX
    • Files need to be re-selected if the file window is closed
    • File paths are truncated, making it difficult to identify correct files
      • Chat window width doesn't accommodate full file names
    • Confusion about time frame/context of previously shown files
    • Natural language file selection doesn't work; must use /include command
      • one tester tried to copy and paste the url with /include command
  2. UI/Visual Issues
    • Similar color between Files icon popup (blue) and chat buttons in dark mode
    • Poor accessible contrast for context references
    • Duo Chat icon lacks GitLab branding, causing confusion
      • One user couldn’t figure out where to find Duo Chat from documentation and thought it was located in Duo Workflow. This suggests we may have future confusion between Duo Chat and Workflow access when launched publicly. User wants to know why the Duo Chat icon doesn’t have GitLab logo: “I would expect the Gitlab logo to open all the things”
  3. File Discovery
    • Difficulty finding files in larger repositories
    • Search functionality not consistently returning expected files
      • One tester couldn't find the file that she recently opened and other files that she had just closed were showing up
    • Confusion about file locations when multiple files share the same name
  4. Response Format
    • Responses sometimes too verbose
    • Need better formatting for readability

Positive feedback

  • One developer who previously avoided AI tools became enthusiastic after seeing the quality of generated tests
  • Potential significant time savings in test writing
  • Effective at complex tasks like debugging import errors

Negative feedback

  • Workflow interruptions due to file selection issues
  • Time lost searching for files in large repositories
  • Extra steps required for basic file inclusion
    • One tester noted they must first selection files, then select the actual file indicating it feels like an extra step

Opportunities

  1. Improve file selection
    • Improve natural language file selection
    • Consider options like "Include all open files"
    • Display full file paths
    • Persist file selection across window closures
  2. Enhance file discovery
    • Optimize search for large repositories
    • Add file path filtering options
    • Auto-suggest including related files (e.g., package.json when generating tests)
  3. UI improvements
    • Increase contrast for better accessibility
    • Adjust window width to show full file paths
    • Distinguish UI elements with different colors in dark mode
  4. Context Awareness
    • Maintain history of commonly used file combinations
  5. Response Formatting
    • Implement collapsible sections for lengthy responses

Instructions for developer testing:

Link to test: https://app.usertesting.com/pp/308dcedf-da2f-40b0-a832-5f2a99ce80bd

You'll find instructions on the link too, but I'll copy them here:\

  1. Setup the latest GitLab Workflow Extension for VS Code and open a repo in VS Code (Currently not available in JetBrains) https://docs.gitlab.com/ee/user/gitlab_duo_chat/#use-gitlab-duo-chat-in-vs-code\\
  2. Pick a task in your workflow that requires multiple files to complete\
  3. Open DuoChat\
  4. Use the slash command /include\
  5. Select the file(s) to add as context (feature only supports features in a git repo, not local files)\
  6. Complete your task using DuoChat and the selected files

As you work through your task please talk through it, narrating your choices, what you expected, what surprised you, etc.

Objective

Conduct unmoderated testing sessions with internal developers to evaluate the effectiveness and user experience of the context feature Duo IDE

Background

We need to gather more data on how developers interact with and utilize the context feature in real-world scenarios. This will help us identify potential issues, understand user workflows, and improve the overall user experience.

Methodology

  1. Create unmoderated tests using UserTesting platform
  2. Recruit internal developers as participants
  3. Have participants use Duo context feature to complete a task in their IDE
  4. Have participants record their screens and think aloud while completing tasks using the context feature
  5. Collect and analyze the recorded sessions

Test Setup

  • Create the test in UserTesting
  • Use links to invite internal users to participate
  • @nicollem to provide test content and structure

Participant Recruitment

  • @bvenker will rally internal developers to participate in the study

Key Areas to Explore

  1. Effectiveness of the context feature in maintaining conversational context
  2. Impact on developer efficiency and context switching
  3. Integration with workflows involving multiple files
  4. Overall user experience and performance

Deliverables

  1. Recorded sessions of developers using the context feature
  2. Summary of findings and insights
  3. Recommendations for feature improvements

Future Considerations

  • Conduct moderated sessions for deeper exploration of specific issues or hypotheses
  • Expand testing to external developers if needed

CC: @katiemacoy @NickHertz

Please update this issue with any additional information or changes to the plan.

Edited by Nicolle Merrill