UX research: issue filter usability testing

Related issues

#26749 (closed) / #25913 (closed)

Hypothesis

• Users will want to multi-select labels using their mouse.

• Users will not instinctively enter a space between labels to trigger the top-level filter options.

• Users will want to clear a selected label with one click.

• Users will not know that they can search for a label by typing.

• Users will use abbreviated words when searching within a dropdown. Subsequently, they will be frustrated at the accuracy of their results.

• Users will want the labels they have selected in the issue filter to be visually represented with colour.

Script

http://bit.ly/2kih0km

Videos

[User 1] (https://www.youtube.com/watch?v=RQGV7y-6uoQ)

[User 2] (https://www.youtube.com/watch?v=wbz-0Kii4jM)

[User 3] (https://www.youtube.com/watch?v=cF0CS-rcpZU)

[User 4] (https://www.youtube.com/watch?v=UF_zfCYgukw)

[User 5] (https://www.youtube.com/watch?v=8c-hFMPLikE)

Results

• Users will want to multi-select labels using their mouse (#26670 (closed)). 2 out of 5 users attempted to multi-select labels using their mouse. User 3: [2:52] (https://youtu.be/cF0CS-rcpZU?t=2m52s), [4:04] (https://youtu.be/cF0CS-rcpZU?t=4m04s), [10:06] (https://youtu.be/cF0CS-rcpZU?t=10m06s). User 4: [5:03] (https://youtu.be/UF_zfCYgukw?t=5m3s).

• Users will not instinctively enter a space between labels to trigger the top-level filter options (#26670 (closed)). 2 out of 5 users did not instinctively enter a space between labels to trigger the top-level filter options. User 1: [8:17] (https://youtu.be/RQGV7y-6uoQ?t=8m17s), User 3: [5:09] (https://youtu.be/cF0CS-rcpZU?t=5m09s).

• Users will want to clear a selected label with one click. (potentially related to #27286 (closed) allowing users to do this may alleviate some of the concerns of #27286 (closed)). No users requested this functionality. Generally, users were happy to select the text, delete it and hit it.

• Users will not know that they can search for a label by typing. 3 users recognised that they can search for a label by typing. User 2: [7:48] (https://youtu.be/wbz-0Kii4jM?t=7m48s). User 3: [6:58] (https://youtu.be/cF0CS-rcpZU?t=6m58s). User 4: [9:26] (https://youtu.be/UF_zfCYgukw?t=9m26s). 2 users did not: User 1: [10:22] (https://youtu.be/RQGV7y-6uoQ?t=10m52s). User 5: Showed very little understanding of the issue filter throughout testing.

• Users will use abbreviated words when searching within a dropdown. Subsequently, they will be frustrated at the accuracy of their results. No users used abbreviated words when searching within a dropdown. Generally, their initial behaviour within a dropdown was to scroll, followed by searching for the full word length.

• Users will want the labels they have selected in the issue filter to be visually represented with colour. No users mentioned the fact that labels were not visually represented with colour within the issue filter, when they were prompted to suggest improvements to the filter.

Additional usability issues found

• 4 users were confused about the syntax they should use within the issue filter. (#27235 (moved)) [Clip 1] (https://youtu.be/RQGV7y-6uoQ?t=15m30s), [Clip 2] (https://youtu.be/wbz-0Kii4jM?t=6m41s), [Clip 3] (https://youtu.be/wbz-0Kii4jM?t=7m4s), [Clip 4] (https://youtu.be/cF0CS-rcpZU?t=2m11s), [Clip 5] (https://youtu.be/cF0CS-rcpZU?t=3m25s), [Clip 6] (https://youtu.be/UF_zfCYgukw?t=9m27s).

• 3 users overlooked the top-level filter options when they first interacted with the issues filter. 1 user commented that he initially didn’t see it as he typed too quickly. (#27238 (moved)) [Clip 1] (https://youtu.be/RQGV7y-6uoQ?t=5m18s), [Clip 2] (https://youtu.be/cF0CS-rcpZU?t=1m55s), [Clip 3] (https://youtu.be/UF_zfCYgukw?t=4m35s).

• 3 users wanted the ability to be able to complete all actions within the issue filter on a mouse OR a keyboard. Currently, a user has to use a combination of mouse and keyboard to search, delete, etc. In order for this to happen, the following would need to be implemented: Keyboard - Ability to use arrow keys to select a label [Clip 1] (https://youtu.be/wbz-0Kii4jM?t=5m41s). (#26840 (closed)) Mouse - Automatically add a space between filters as they are applied (as discussed above) (#26670 (closed)) Implement a CTA for search. It wasn’t always clear to users that they had to press ‘Enter’ to complete an action. (#27286 (closed)) [Clip 2] (https://youtu.be/UF_zfCYgukw?t=6m8s), [Clip 3] (https://youtu.be/UF_zfCYgukw?t=8m16s), [Clip 4] (https://youtu.be/UF_zfCYgukw?t=10m23s), [Clip 5] (https://youtu.be/8c-hFMPLikE?t=7m36s), [Clip 6] (https://youtu.be/8c-hFMPLikE?t=8m22s).

• 3 users tried to trigger the top level filter menu by clicking on the filter icon. (#27239 (closed)) [Clip 1] (https://youtu.be/cF0CS-rcpZU?t=5m19s), [Clip 2] (https://youtu.be/UF_zfCYgukw?t=9m), [Clip 3] (https://youtu.be/8c-hFMPLikE?t=5m59s), [Clip 4] (https://youtu.be/8c-hFMPLikE?t=7m26s).

• 1 user tried to create a label which already existed on the new/edit issue page. There was no notification to inform the user that the label already existed, therefore it wasn’t clear to him what was happening. (#26945 (closed)) [Clip 1] (https://youtu.be/UF_zfCYgukw?t=11m22s).

• 1 user commented that he disliked and struggled to use nested scrollbars. (#27289 (moved)) [Clip 1] (https://youtu.be/RQGV7y-6uoQ?t=14m16s) .

• 1 user struggled to understand the Label dropdown menu which appears when a checkbox is marked. The heading of this dropdown is ‘Filter by label’, he assumed his search had been filtered when he selected a label. When, in fact, he’d added a label. (#27352 (closed)) [Clip 1] (https://youtu.be/8c-hFMPLikE?t=6m29s), [Clip 2] (https://youtu.be/8c-hFMPLikE?t=11m35s), [Clip 3] (https://youtu.be/8c-hFMPLikE?t=13m2s).

Inconclusive usability issues (more research required)

• Investigate users’ understanding of iconography (#27288 (closed)) [Clip 1] (https://youtu.be/RQGV7y-6uoQ?t=54s), [Clip 2] (https://youtu.be/UF_zfCYgukw?t=1m27s), [Clip 3] (https://youtu.be/UF_zfCYgukw?t=4m10s), [Clip 4] (https://youtu.be/UF_zfCYgukw?t=11m).

• Autocompletion of keywords (#27290 (closed)) [Clip 1] (https://youtu.be/RQGV7y-6uoQ?t=5m25s).

• 2 users didn’t realise that labels could be applied to issues from the issues page. (#27290 (closed)) [Clip 1] (https://youtu.be/RQGV7y-6uoQ?t=13m47s), [Clip 2] (https://youtu.be/UF_zfCYgukw?t=10m40s).

• Of the 2 users who went through to the issue’s detail page to apply a label, both hesitated when they were presented with the page. It wasn’t clear to them how to apply a label. (#27290 (closed)) [Clip 1] (https://youtu.be/RQGV7y-6uoQ?t=13m52s), [Clip 2] (https://youtu.be/UF_zfCYgukw?t=10m53s).

Bugs

• Search fails, an issue should have been found (unable to reproduce) [Clip 1] (https://youtu.be/RQGV7y-6uoQ?t=15m41s).

• Some of the filters are removed as the page refreshes (unable to reproduce) [Clip 1] (https://youtu.be/wbz-0Kii4jM?t=5m29s).

• The same label appears multiple times within the labels filter (fixed - !8710 (merged)) [Clip 1] (https://youtu.be/cF0CS-rcpZU?t=3m7s).

• If the same label is applied twice, should the issue filter still return issues? (#27293 (closed)) [Clip 1] (https://youtu.be/UF_zfCYgukw?t=5m36s).