Click Testing - Tests 1-5

Progress

  • Establish variants
  • Set up click tests
  • Write email to accompany click tests
  • Distribute email to a test sample of users
  • Review answers and make any necessary amendments
  • Distribute email to remaining users
  • Purchase Amazon vouchers, conduct prize draw and contact winners
  • Analyse results
  • Write up findings
  • Update relevant issues in the CE project

What's being tested?

No issue at present, MR: https://gitlab.com/gitlab-org/gitlab-ce/issues/34261 - Do users know how to move an issue between projects?

https://gitlab.com/gitlab-org/gitlab-ce/issues/37399 - Is the new project page overwhelming? Are the tabs discoverable? Should we iterate towards Pedro's 'project flow' design?

https://gitlab.com/gitlab-org/gitlab-ce/issues/38454 - Is it easy to distinguish between an issue and merge request?

https://gitlab.com/gitlab-org/gitlab-ce/issues/35740 - What impact will changing the colour of the sidebar have on a user's ability to find content located in the sidebar?

Findings

Do users know how to move an issue between projects?

Accuracy

  • Default (Move Issue CTA) - 52.5%
  • Old location (Editing the issue) - 50%

Speed

  • Default (Move Issue CTA) - 18.7 seconds
  • Old location (Editing the issue) - 16.8 seconds

Conclusion

Accuracy for both locations is quite low, it appears moving an issue whether using the new 'Move issue' CTA or via editing an issue is not intuitive. For users which were shown the new 'Move Issue' CTA variant, I analysed how many users selected the old location, 27.5% of users did at an average speed of 22.8 seconds.

The implementation of the 'Move Issue' CTA is still relatively new and users might still be getting to grips with its location. My recommendation is to monitor the situation and re-run this test in a month to see whether the stats have improved. If the stats haven't improved, I'd be inclined to suggest testing another location for the 'Move Issue' CTA. We should also be aware that any changes we make to the sidebar (e.g. https://gitlab.com/gitlab-org/gitlab-ce/issues/35740) may affect the discoverability of the CTA.

Is the new project page overwhelming? Are the tabs discoverable? Should we iterate towards Pedro's 'project flow' design?

Creating a blank project - Accuracy

  • A (Default) - 75%
  • B (Tabbed) - 74%
  • C (Wizard) - 80%

Creating a blank project - Speed

  • A (Default) - 10.4 seconds
  • B (Tabbed) - 11.7 seconds
  • C (Wizard) - 7.7 seconds

Importing a GitHub project - Accuracy

  • A (Default) - 82%
  • B (Tabbed) - 94%
  • C (Wizard) - 100%

Importing a GitHub project - Speed

  • A (Default) - 6 seconds
  • B (Tabbed) - 5.7 seconds
  • C (Wizard) - 5.2 seconds

Conclusion

Tabbed: A tabbed design did not hinder discoverability although users were still overwhelmed by the options available on the 'Blank Project' tab hence the lower accuracy and speed. If we could iterate on / simplify the 'Blank Project' tab, the design could be a good interim solution.

Wizard: Faired best in both tasks. I think this is the design we should be iterating towards.

Is it easy to distinguish between an issue and merge request?

Users who were shown the top of a merge request

  • 62.5% of users correctly identified that they were looking at a merge request.
  • 32.5% of users thought they were looking at an issue.
  • 5% of users weren’t sure what they were looking at and chose to pass on the question.
  • On average it took users 27.4 seconds to answer the question.

Users who were shown the bottom of an issue

  • 63% of users correctly identified that they were looking at an issue.
  • 22% of users thought they were looking at a merge request.
  • 15% of users weren’t sure what they were looking at and chose to pass on the question.
  • On average it took users 28.6 seconds to answer the question.

Conclusion

Whilst the majority of users could correctly identify what they were looking at, I still think the results are quite low and could be improved upon. Likewise, with the time it took users to identify what they were looking at - almost 30 seconds in each use case is incredibly slow. I’d propose making issues and MRs more visually distinctive from one another, whether through the use of colour or another method - I’m happy to test multiple solutions.

What impact will changing the colour of the sidebar have on a user's ability to find content located in the sidebar?

Accuracy

  • A - 90%
  • B - 81%

Speed

  • A - 10.6 seconds
  • B - 12.1 seconds

Conclusion

A white sidebar performed worse than our existing sidebar colour. Users also had a higher propensity to click the ‘Edit’ CTA on the issue description when they were shown the white sidebar, suggesting that users were less likely to align the sidebar content with the page content.

Edited by Sarah Jones