1.3 Global vs contextual navigation testing
Follow-up testing to: #3 (closed) and #5 (closed)
Related to: #6 (closed)
Prototypes:
Prototype A: https://framer.cloud/fyEvF/index.html
Prototype B: https://framer.cloud/jveno/index.html
Script
Users
User Number | Job | Existing GitLab User? | GitLab Features Used | Existing GitHub User? | Prototype Shown |
---|---|---|---|---|---|
1 | Development Team Lead | Yes | Merge Requests, Continuous Integration, Wiki | No | A |
2 | DevOps | Yes | Merge Requests, Continuous Integration, Issue Management | Yes | B |
3 | Executive (VP of Eng., CTO, CIO, etc) | Yes | Merge Requests, Continuous Integration, Issue Management, GitLab Pages | Yes | B |
4 | Full-stack web developer | Yes | Merge Requests, Continuous Integration, Issue Management, Wiki, Time Tracking. | Yes | A |
5 | Full-stack web developer | Yes | Merge Requests, Wiki. | No | B |
6 | Back-end web developer | Yes | Merge Requests, Continuous Integration, Issue Management, Wiki | Yes | A |
7 | DevOps Lead | No | None | Yes | A |
8 | Quality Assurance Engineer | No | None | Yes | B |
9 | Systems Administrator | Yes | Merge Requests, Continuous Integration, Wiki | Yes | A |
10 | DevOps | No | None | Yes | B |
11 | Student | No | None | Yes | A |
12 | Full-stack web developer | Yes | Merge Requests, Issue Management | Yes | B |
Videos
User 1 - http://bit.ly/2plQ8FP
User 2 - http://bit.ly/2psV6AH (Restrictive access, user does not wish to share the video publicly)
User 3 - http://bit.ly/2oGYTWi
User 4 - http://bit.ly/2oL7JTX
User 5 - http://bit.ly/2piVqiz
User 6 - Recording failed. My notes: http://bit.ly/2p4AwVY I managed to salvage some footage: http://bit.ly/2qfeYDQ (Restrictive access, user does not wish to share the video publicly)
User 7 - http://bit.ly/2qiFYSU
User 8 - http://bit.ly/2pqlwSL
User 9 - http://bit.ly/2qlXmX1 (Restrictive access, user does not wish to share the video publicly)
User 10 - http://bit.ly/2ppfa4m
User 11 - http://bit.ly/2qmc1Bw
User 12 - First clip: http://bit.ly/2pce2Cp Second clip: http://bit.ly/2qeWJPz (Restrictive access due to personal content)
Findings
Average time taken per task
Prototype A
- | Task 3 | Task 4 | Task 5 | Task 6 | Task 7 | Task 8 | Task 9 | Task 10 | Task 11 | Task 12 | Task 13 | Task 14 | Task 15 | Task 16 |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
User 1 | 7 | 4 | 7 | 66 | 9 | 36 | 34 | 36 | Withdrawal | 30 | 76 | 17 | 26 | 62 |
User 4 | 9 | 6 | 3 | 15 | 6 | 8 | 1 | 8 | 24 | 12 | 20 | 2 | 11 | 10 |
User 6 | 9 | 6 | 7 | 160 | 69 | 34 | 9 | 7 | 80 | 11 | 15 | 4 | 52 | 13 |
User 7 | 6 | Withdrawal | 5 | 49 | 124 | 18 | 5 | 17 | 35 | 12 | 8 | 2 | 9 | 13 |
User 9 | 10 | 8 | Withdrawal | 36 | 71 | 38 | 18 | 24 | Withdrawal | 24 | 89 | 11 | 22 | 22 |
User 11 | 7 | 6 | 64 | Withdrawal | 117 | Withdrawal | 6 | 95 | 70 | 10 | Withdrawal | 3 | 14 | Withdrawal |
Average time taken: | 8 (6,10) | 6 (4,8) | 9 (2,37) | 49 (17,143) | 40 (10,160) | 23 (10,53) | 7 (2,27) | 21 (7,58) | 47 (19,116) | 15 (9,24) | 28 (8,102) | 5 (2,12) | 19 (9,37) | 19 (8,46) |
Prototype B
- | Task 3 | Task 4 | Task 5 | Task 6 | Task 7 | Task 8 | Task 9 | Task 10 | Task 11 | Task 12 | Task 13 | Task 14 | Task 15 | Task 16 |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
User 2 | 11 | 32 | Withdrawal | 39 | 32 | 33 | 32 | 11 | Withdrawal | 24 | Withdrawal | 4 | 46 | Withdrawal |
User 3 | 11 | 6 | 10 | 7 | 8 | 18 | 8 | 18 | Withdrawal | 22 | 25 | 40 | 28 | 36 |
User 5 | 8 | 4 | 22 | 16 | 12 | 10 | 5 | 14 | 17 | 30 | 10 | 3 | 16 | 14 |
User 8 | 8 | 2 | 3 | 59 | Withdrawal | Withdrawal | Withdrawal | 34 | Withdrawal | 20 | Withdrawal | 3 | 15 | 20 |
User 10 | 22 | 20 | 25 | 27 | 65 | 104 | 5 | 15 | 71 | 187 | 83 | 7 | 21 | 20 |
User 12 | 6 | 18 | Withdrawal | 47 | Withdrawal | 20 | 10 | 33 | 120 | 23 | Withdrawal | 1 | 9 | 6 |
Average time taken: | 10 (6,16) | 9 (3,28) | 11 (2,53) | 26 (11,60) | 21 (5,96) | 26 (9,78) | 9 (4,24) | 19 (12,31) | 53 (4, 637) | 33 (14,82) | 27 (2, 377) | 5 (1,17) | 20 (11,36) | 16 (7,37) |
- Time is in seconds.
- Average time taken is calculated using geometric mean.
- Withdrawals have been excluded from the average time taken.
- The confidence interval for each task is shown in brackets. The confidence level is 95%.
What does the average time taken tell us?
-
Users who interacted with Prototype A took almost twice as long to complete task 6 which was to locate issues. This was the first task that asked users to interact with the contextual navigation at a project level. This suggests that users of Prototype A may have been slightly more disorientated than users of Prototype B at this point during the testing. However, later tasks which asked users to find other content within the contextual navigation at a project level show that Prototype A and B performed similarly. Indicating that users of Prototype A became more familiar with the contextual navigation through use and ended up with a comparable understanding to users testing Prototype B.
-
Users of Prototype A also took almost as twice as long to complete task 7 which was to locate the issues icon, although overall users of Prototype A generally had a greater success rate of completing this task. This was the first task that asked users to interact with their personal navigation. Despite Prototype A having a higher completion rate, the task time was affected due to more users taking an indirect path to the issues icon. Whereas the majority of users interacting with Prototype B who did successfully complete the task took a direct path. This indicates that users of Prototype A were confused about the content they could find within the contextual navigation. This correlates with the email feedback from one user (who used Prototype A) in relation to
+ icon
, which he also expected to find within the contextual navigation:
The problem was that, for almost everything related to issues, I would first find the noun I am working with (issues) and find that section in the navigation (which I knew was in the sidebar). Then, I would search that area of the page for the verb (create/add). For things like listing/viewing issues or anything other than create, this would be true. However, for creating an issue, I first had to think of the verb (create), find that menu item in the top navigation, and then find the noun (issue) in that section. This reversal (find verb -> find noun) caused confusion and made it difficult for me to remember that the flow is reversed only for times when I wanted to create something. All other times I had to go to the issue section first.
- Users of Prototype B took over twice as long to complete task 12, which was to find how many subgroups the group of 'customers' had. This is because users of Prototype A had by this point in the testing a much greater understanding of what they could expect to find in the global navigation vs contextual navigation. Almost all users of Prototype A took a direct path to their subgroups and did so via the global navigation, resulting in a faster task completion time. Users of Prototype B took varied, indirect paths.
Usability issues and recommendations
Area Affected | Problem | Number of Affected Users | Recommendations / further comments | Related issue number |
---|---|---|---|---|
+ icon | Users were slow to adopt the + icon. The first task in which users could have interacted with the + icon was task 3. Only 2 out of 12 users tested chose to use the + icon on this task (Prototype B: 2 users). | Prototype A: 6/6, Prototype B: 4/6 | It's surprising that 10 of the users we tested with used GitHub yet this did not seem to impact on their recognition or use of the + icon. I think it would be interesting to conduct some usability testing on GitHub with 'power users'. Low usage on GitHub could indicate that users don't need this functionality whereas high usage may indicate a problem with our implementation of the icon. | https://gitlab.com/gitlab-org/gitlab-ce/issues/23603 |
+ icon | 3 users failed to interact with the + icon throughout testing | Prototype A: 1/6, Prototype B: 2/6 | ||
+ icon | 2 users who did interact with the + icon initially assumed it was contextual - they navigated to a project before clicking it to create an issue. Similar behaviour has been seen in previous usability testing. | Prototype A: 1/6, Prototype B: 1/6 | https://gitlab.com/gitlab-org/gitlab-ce/issues/23603 | |
Breadcrumb | On Tasks 1 and 2, no user correctly identified that they were inside of a subgroup. This was due to the fact that subgroups were a new GitLab feature when the testing took place and users were not yet familiar with the concept of subgroups. However, some users did spot the breadcrumb on these tasks and were able to correctly identify that they were within the project of 'stack-story'. On both prototypes, 3 out of the 6 users tested could correctly identify their location. Whereas the remaining 3 users for each prototype could not. | Prototype A: 3/6, Prototype B: 3/6 | ||
Breadcrumb | 7 users did not interact with either breadcrumb during the testing. | Prototype A: 4/6, Prototype B: 3/6 | ||
Breadcrumb | At the end of the testing, where a user had not mentioned or interacted with a breadcrumb, I prompted them to discuss the relationship between 'stack story', 'customers' and 'organisation'. Even though this was a deliberately leading request, 2 users still overlooked the breadcrumb. | Prototype A: 1/6, Prototype B: 1/6 | Looking at the results, there does not seem to be a clear breadcrumb winner. Even taking users' personal opinions / comments into account, the prototypes received similar feedback. | |
Creating a new project | 3 users commented that they thought they could add members to a project from the project creation screen. One non-GitLab user further explained that this was something GitHub currently does not do. This is frustrating for her as she generally works on projects with other people. | 3/12 | ||
Snippets | Generally, the success rate for task 5 was very good (9/12). However, some users commented that although they felt they had successfully completed the task, they weren't sure what a 'snippet' actually was. I also felt some users failed this task because they didn't necessarily know what terminology they should be looking for in the interface. | 5/12 | Perhaps we could explain to users what a snippet is during onboarding or within the interface when the user is yet to create a new snippet (empty state). | https://gitlab.com/gitlab-org/gitlab-ce/issues/20847 |
Dashboard | After completing task 6, two users commented that they weren't sure where they were currently located (within My Dashboard). | Prototype A: 1/6, Prototype B: 1/6 | ||
Contributed Projects | 5 users failed to find their 'Contributed Projects'. Of the 7 users who did find their 'Contributed Projects', 6 took an indirect path. | 11/12 | Most users instinctively went to 'Your Projects' before ruling it out. Could we let users filter by 'Contributed Projects' on this page? | https://gitlab.com/gitlab-org/gitlab-ce/issues/32810 |
Explore Projects | 4 users failed to find 'Explore Projects'. Of the 8 users who did find 'Explore Projects', 4 users took an indirect path. | 8/12 | Users felt both prototypes only showed them a "view of my own things" and when questioned said they were "looking for a way out of my personal account". User 9 in particular summed this up well, by explaining that he felt the sidebar was specific to his account as it contains language appertaining to him such as 'Your Projects'. 'Explore Projects' is related to other peoples' projects, therefore he didn't expect to find the two situated so closely together. | |
Explore Projects / Tanuki | When attempting to find 'Explore Projects', 7/12 users interacted with the tanuki during the task. In previous testing, users commented that they felt it would take them to a dashboard view. However, during this round of testing, this assumption changed. Users thought it would direct them out of their personal account and to a page which would show them today's trending projects (similar to https://github.com/explore). | 7/12 | ||
Group you are not a member of | There were still some users who used process of elimination, clicking back and forth between ‘Your Groups’ and ‘Explore Groups’ to complete this task. Also, despite providing the correct answer, this was followed by phrases such as "I think", "I suspect", etc showing that users are still not fully confident in their answer. | 6/12 | ||
Sidebar (contextual navigation) | Generally, the majority of users seemed to understand that the sidebar was contextual after completing 6-7 tasks. This is an improvement from previous research which showed that users could complete a whole usability testing session without understanding that the sidebar is contextual. However, I feel it's important to note that both the behaviour and accompanying comments made by existing GitLab users, indicate that it may be difficult for them to transition to the sidebar. Existing GitLab users still instinctively look to the centre of their screen for content, for example User 12 struggled to find 'Issues' because he was familiar with its current location and User 5 discussed how by referencing the global navigation, he subsequently didn't always spot that the contents of the sidebar had changed, as his attention was focussed elsewhere on the screen. We're forcing users to change a behavioural pattern that they may have had for a few years which might be met with some resistance. | @cperessini has already mentioned slowly transitioning users to the new sidebar and I think that's a really good idea. I also think that whilst we've agreed that it feels like users are completing tasks faster - we may actually want the research to support that which would mean running the same script on GitLab's existing navigation. Finally, maybe there's still improvements that can be made visually to help users shift their gaze to the left sidebar? |
#5 (closed))
Progress since last usability testing session (| Area Affected | Observation | Affected Users | Related issue number | |---------------------- |-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |----------------------------------------------------------------------------------------------------------------- |--------------------------------------------------------------------------------------------------------------------------------------------- |---------------------------------------------------------- |------- |--- ||Global Navigation | Generally the Global navigation tested well with all users. In particular 'recent projects' was used frequently as a means of navigating into 'stack-story' quickly. Users' comments were also complimentary about it. | | | Project 'Owner' | Almost all users understood what the label of 'owner' meant on the 'Your Projects' screen. | 10/12 | https://gitlab.com/gitlab-org/gitlab-ce/issues/27742 | | Issues / Todos Icon | The tooltips seemed to work better on these prototypes and subsequently users utilised them to identify the icons. No users seemed particularly bothered that they had to use the tooltips to identify the icons. Only 1 user commented that he didn't understand why assigned issues used a # symbol. | 11/12 | | Interface as a whole | No users commented that the interface looks like a ‘“tablet”, “mobile” or “responsive view”. First impressions of the issues detail page were that it was busy and overcrowded, however, generally that perception changed during the testing once users established the significance of the content. | 12/12 |
cc @cperessini @pedroms @dimitrieh @tauriedavis @hazelyang @mydigitalself