Skip to content

Project & Group Level Tree Testing (Contextual Navigation)

Progress

  • Write tasks for the tree test.
  • Enter tasks into the testing software.
  • Enter project and group level contextual navigation items into the testing software.
  • Write email to accompany tree tests.
  • Distribute tree test to a test sample of users.
  • Review answers and make any necessary amendments.
  • Distribute email to remaining users.
  • Purchase Amazon vouchers, conduct prize draw and contact winners.
  • Cleanse data.
  • Analyse results.
  • Write up findings.
  • Create new issues as required.
  • Update relevant issues in the CE project.

Research Questions

  • How much of an improvement is the recommended structure for the contextual navigation (as detailed in #35 (closed))?
  • Do the differences in how content is organised at a project and a group level cause users confusion?
  • Are there any further improvements we can make to the recommended structure based on the results of the tree test?
  • Do users confuse the content of Graph and Charts?
  • What small, iterative changes can we make to the sidebar navigation?

Methodology

  • Test the existing contextual navigation at a project and group level.
  • Test the recommended structure for the contextual navigation at a project and group level (as detailed in #35 (closed)).
  • Ensure the project and group contextual navigation are tested together, rather than in isolation of one another.
  • Ensure there is a task which will test whether users get confused between the content of Graph and Charts.

Results

Recommended structure for the contextual navigation

Project Level

  • Project
    • Details
    • Activity
  • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Graph
    • Compare
    • Locked Files
    • Snippets
  • Registry
  • Issues
    • List
    • Boards
    • Labels
    • Service Desk
    • Milestones
  • Merge Requests
  • CI/CD
    • Pipelines
    • Jobs
    • Schedules
    • Environments
    • Charts
  • Statistics
    • Cycle Analytics
    • Contributors
    • Members
    • Charts
  • Wiki
  • Settings (if applicable)

Group Level

  • Overview
    • Details
    • Activity
  • Issues
    • List
    • Boards
    • Labels
  • Merge Requests
  • Planning
    • Epics
    • Milestones
    • Roadmap
  • Statistics
    • Contribution Analytics
    • Members
  • Settings (if applicable)

Recommendation 1

At a project level, change: Overview/Details to Project/Details

Why?

  • +7% increase in users successfully completing the task.
  • In comparison to the existing structure, users were +3 seconds quicker to complete the second task they were served. Emphasizing the differences at a project and group level have helped users to identify where they are located within GitLab.

Recommendation 2

At a group level, move Issues/Milestones to Planning/Milestones

Why?

  • +5% increase in users successfully completing the task.
  • Very little difference in how long it took users to select the correct path (Proposed:14 seconds vs Existing:15 seconds). This was surprising as users taking the existing tree test had more experience of using Milestones (Proposed:61% vs Existing:86%) and Groups (Proposed:83% vs Existing:95%). Combined with the fact that users of the existing structure should, in general, be more familiar with the location of Milestones. You would expect users to be much quicker to identify the correct path on the existing structure.
  • 26% of users who completed tasks on the existing tree test structure incorrectly selected the Epics List or Roadmap as their answer, highlighting that user intent is to see Epics, Roadmap and Milestones grouped together (or perhaps even consolidated into one view). Users did not typically look for this grouping within the category of Issues.

Recommendation 3

At a project level, move Repository/Charts to Statistics/Charts

Why?

  • Users who completed tasks on the existing structure had more experience of using Charts (Proposed:32% vs Existing:46%), despite this the results for both success (Proposed:53% vs Existing:54%) and time taken (Proposed:13 seconds vs Existing:12 seconds) were almost identical. We need more data to understand the true value of moving Charts to Statistics, however, we've identified that it's a low-risk change to release and it would make sense to measure the impact post-release.
  • Also see response to: Do users confuse the content of Graph and Charts? (below, under Responses to research questions).

Recommendation 4

At a project level, move Overview/Cycle Analytics to Statistics/Analytics

Why?

  • +5% increase in users successfully completing the task.

Recommendation 5

At a group level, when released, add Roadmap to Planning/Roadmap

At a group level, move Epics/List to Planning/Epics

Why?

  • Users had a higher success rate when completing the Roadmap task on the proposed structure (+4% increase in success).

  • Epics performed poorly on both the proposed and existing structure, however, this is to be expected as the accompanying questionnaire to the tree tests showed that only a small number of users were utilising this functionality (Epics had been released for approx 1 month when the testing took place). Settings was the most common answer on both tree tests. This may potentially indicate that users feel they need to toggle on Epics in order to view them within GitLab.

  • Based on recommendation 2, we know users expect to find Epics, Roadmap and Milestones grouped together. The success rate for both Roadmap and Milestones was higher when placed within the category of Planning, therefore it is a low-risk change to move Epics to this category as well and measure the impact post-release.

Responses to research questions

How much of an improvement is the recommended structure for the contextual navigation (as detailed in #35 (closed))?

This is detailed in each recommendation.

Do the differences in how content is organised at a project and a group level cause users confusion?

In the case of the recommendations, no the differences in how content is organised at a project and group level do not cause users confusion.

On some tasks, the differences in how content is organised at a project and group level did cause users confusion. For example, with Merge Requests, users typically followed a similar path at a project and group level, so when those paths were different to one another, users were less successful on the second task they were served.

Are there any further improvements we can make to the recommended structure based on the results of the tree test?

Not so much further improvements, but the results of the tree test showed that a hybrid of the existing and proposed structure worked best, which is reflected in the recommended structure for the contextual navigation (as above).

Do users confuse the content of Graph and Charts?

Users had a higher propensity to confuse the content of Graph and Charts when the items were placed within the same category of Statistics. The success rate for Graph was much higher when located within Repository (Existing) showing that, for now, it should continue to reside there.

We identified in recommendation 3 that Charts could potentially work well in the Statistics category and by keeping Graph within Repository, user confusion is lowered, strengthening the positioning of Charts within Statistics.

What small, iterative changes can we make to the sidebar navigation?

Iteration 1: Recommendation 1

Iteration 2: Recommendation 2 and Recommendation 5.

Iteration 3: Recommendation 3 and 4.

Edited by Sarah Jones