Design Evaluation: Issue "Closed" Status Icon

What’s this issue all about?

Investigate how users perceive the current icons used for a "closed" issue.

• We are testing the existing experience, with the “closed” issue icon as the only variable.
• We will see how users perform on the test and identify whether issue+checkmark is a good fit before testing any other proposed icons.

What questions are you trying to answer?

Which icon do users prefer to represent a "closed" issue?

What assumptions do you have?

• Some users find the current Issue "closed" status icon confusing.
• "Do not enter" sign should not represent "closed" issue

What decisions will you make based on the research findings?

We will decide on keeping the current icons or updating them.

Methodology

Preference testing via UsabilityHub sent to GitLab Twitter.

Progress

• Choose 2-3 icons for preference test. [Deadline: Fri Feb 15th]
• Create a preference test in UsabilityHub [Deadline: Mon Feb 18th]
  • Preview/test study to double-check flow [Deadline: Mon Feb 18th] --> https://usabi.li/do/s/2112e6
• Create click test with variation set [Deadline: Fri Feb 15th]
  • Create mockups for click test [Deadline: Fri Feb 15th]
• Advertise study on GitLab Twitter [(Target) Deadline: week of March 4th] --> https://gitlab.com/gitlab-com/marketing/corporate-marketing/issues/363
• 2nd round of advertising on social media to gather more responses [Deadline: Fri Mar 15th]
• Review results and share! [Deadline: Weds Mar 20th]

added Plan [DEPRECATED] UX Research Backlog UX research labels

changed milestone to %11.9

changed the description

marked the checklist item Choose 2-3 icons for preference test. [Deadline: Fri Feb 15th] as completed

marked the checklist item Create a preference test in UsabilityHub [Deadline: Mon Feb 18th] as completed

Hi @evhoffmann! Would it be possible to schedule a tweet for a preference test on GitLab Twitter for the week of Feb 18th?

Copy would be something like:

Do you have a minute? We've created a quick test to get some feedback on GitLab Issues: https://usabi.li/do/s/2112e6

P.S. Tweets like this may happen more often in the future (if okay with Marketing). Please let me know if there is a certain process for requesting

changed the description

@victorwu If you'd like to try out the full test, here's the link to a preview: https://app.usabilityhub.com/preview/deb0465f0dfa.

I also created a version of the test to collect answers from GitLabbers only (https://usabi.li/do/57962b6389ee/38a0). Just to see how our answers compare to other users. I'll share it in Slack!

Looks great @katokpara . Thanks for putting this together so quickly. I noticed there was another question regarding issue titles in the survey. Was that intentional to get a user to do a little more work and related to some other research? Just curious.

marked the checklist item Choose 2-3 icons for preference test. [Deadline: Fri Feb 15th] as incomplete

@victorwu No problem! And yes those other questions are intentional and related to https://gitlab.com/gitlab-org/gitlab-ce/issues/47466! I just included them since it's a smaller question that I haven't been able to look into yet.

Thanks for taking a look at the test

Sure thing @katokpara . Of course beware if I polluted any data that you can probably pick out easily from timestamps and such.

Would it be possible to schedule a tweet for a preference test on GitLab Twitter for the week of Feb 18th?

Absolutely! Will add a screenshot here when it's scheduled.

Tweets like this may happen more often in the future (if okay with Marketing). Please let me know if there is a certain process for requesting

Totally fine! I always have to mind the schedule to juggle what else is being promoted - as long as we don't have these tweets going out on consecutive days, it should be fine. Every other week or so at most would probably be my suggestion to avoid saturating the audience.

Process-wise, feel free to either ping me in the issue like you did, or open an issue https://gitlab.com/gitlab-com/marketing/corporate-marketing/issues with the date, details, and link to the research issue.

Totally fine! I always have to mind the schedule to juggle what else is being promoted - as long as we don't have these tweets going out on consecutive days, it should be fine. Every other week or so at most would probably be my suggestion to avoid saturating the audience.

Thanks @evhoffmann! And I totally agree - anything more than every other week would probably become less intriguing to the audience.

Hi @evhoffmann We're continuing to discuss this issue (https://gitlab.com/gitlab-org/gitlab-ce/issues/57672#note_141699036) so I'll reach out to you again for when we should actually tweet the study

marked the checklist item Choose 2-3 icons for preference test. [Deadline: Fri Feb 15th] as completed

marked the checklist item Preview/test study to double-check flow [Deadline: Mon Feb 18th] --> https://usabi.li/do/deb0465f0dfa/b94d (external) as completed

changed the description

changed the description

@annabeldunstone After giving it some more thought, we need to get a better sense of how well users understand the icons, in and out of context.

These are the 4 "closed" issue icons we're testing:

Existing	Proposed Replacement	Additional Suggestion	Additional Suggestion

Would you be able to make 4 mockups that include one of each "closed" icon in a view similar to this screenshot:

This is so users will be able to see how the icons would actually look in the Related Issues widget.

assigned to @annabeldunstone

@katokpara Sure! Assigning myself as well so I don't lose track of this

changed the description

@katokpara Here you go; let me know if you need anything else!

Thank you so much @annabeldunstone! I think these will work really well. I'll let you know if anything else comes up

marked the checklist item Create mockups for click test [Deadline TBD] as completed

marked the checklist item Create click test with variation set [Deadline: Fri Feb 15th] as completed

changed the description

@annabeldunstone @katokpara do you think the issue + checkmark icon should be tested alongside a green issue icon for the open state instead of the green circle? All the other ones use circles for both states, so I think using rectangles for both states in this case would make sense.

@cperessini I think it's best to test the issue + checkmark with the green circle since they would appear that way in reality (assuming we don't change the open state to match the new closed state).

Do we have a definition for what the Open circle is supposed to mean? Does it come from the Do not enter sign with the bar removed? Or maybe does it relate to the concept of openness in an empty circle? I think it would be useful to understand what the circle means to users to see if it makes sense to use it as the base shape for all status icons. (source: https://gitlab.com/gitlab-org/gitlab-ce/issues/57672#note_141699036)

However, I do plan to address your previous question, by including a task where users see the green circle and are asked to identify the status of the issue. Then a followup question will ask them to describe why they think the circle icon relates to the status they selected.

I think it's best to test the issue + checkmark with the green circle since they would appear that way in reality (assuming we don't change the open state to match the new closed state).

@katokpara Ah, of course. I was actually thinking that if we went with that approach, we would replace the circle with an issue/epic/MR icon depending on the object, but I forgot to mention that in my comment. I don't know if that's what @annabeldunstone had in mind, though.

However, I do plan to address your previous question, by including a task where users see the green circle and are asked to identify the status of the issue. Then a followup question will ask them to describe why they think the circle icon relates to the status they selected.

That sounds great, thanks! That will be very valuable information, I think

@annabeldunstone we've also decided to test the visibility of the "Related Issues" widget and investigate how well users can distinguish it from the Related MRs widget.

It'd be awesome if you could either update the 4 mockups to include a Related MR widget or create new mockups similar to the screenshot below. Whichever makes more sense Thank you!!

I think after this we should be almost ready to send it out

I was actually thinking that if we went with that approach, we would replace the circle with an issue/epic/MR icon depending on the object, but I forgot to mention that in my comment. I don't know if that's what @annabeldunstone had in mind, though.

@cperessini Yes, that is what I had in mind too! So for open MRs we would use git-merge and for open epics we would use epic. For closed, we would just layer a checkmark over each icon.

The only issue next is the issues icon looks like it's denoting issues, plural. For a single issue, will that be confusing?

@annabeldunstone In that case, you could also create a mockup for issue + checkmark paired with the green issue icon!

So the mockups I need are:

1. Blue Issue + checkmark, green issue icon, related MR widget
2. Blue Issue + checkmark, current open issue icon, related MR widget with open/closed/merged states
3. Blue Do Not Enter icon (current closed icon), current open icon issue, related MR widget with open/closed/merged states
4. Blue Filled Circle icon , current open issue icon, related MR widget with open/closed/merged states
5. Blue "Ban" icon , current open issue icon, related MR widget with open/closed/merged states

@cperessini Can you please share the location of your icon designs? https://gitlab.com/gitlab-org/gitlab-ce/issues/57672#note_141699036

@annabeldunstone I hacked the icons together and threw the file away after posting my comment, sorry!

You may want to create the icons properly, but this is what I did myself:

• For the closed epic icon, I took the epic icon and pasted the checkmark from issue-created with a 8px diameter circle behind.
• For the closed MR icon, I took the MR icon and covered the top part with a white rectangle and added the cross from the unapproval icon.

@katokpara I'm not going to be able to get to this today; will try to upload the screenshots tomorrow

@katokpara Just to make sure I'm on the right track- is this what you were thinking for mockup 1?

@annabeldunstone yes it is, thank you!

cc @katokpara

@annabeldunstone Thank you!!

A few fixes:

• All the screenshots should only have the existing MR icons. Our only variable is the "closed" issue icon. Additionally, could you add a red closed MR in the related MR widget as well?

• There should be two closed issues in the Existing closed icon screenshot (the 4th one) so that it's consistent with the other screenshots.

Sorry for all the little changes! Hopefully after this we'll be good to go

changed the description

@katokpara Round 3

1	2	3	4

@annabeldunstone Perfect! Thank you

Hi @evhoffmann This study is ready to be scheduled! Here's the updated link: https://usabi.li/do/s/2112e6.

added In Progress label and removed UX Research Backlog label

added to epic &582 (closed)

changed the description

marked the checklist item Advertise study on GitLab Twitter [(Target) Deadline: week of March 4th] --> https://gitlab.com/gitlab-com/marketing/corporate-marketing/issues/363 as completed

Update: This test is scheduled to go out on Twitter at 1pm PT today (Mar 5th) and tomorrow (Mar 6th) on Facebook at 5am PT.

https://gitlab.com/gitlab-com/marketing/corporate-marketing/issues/363#note_147550377

Thanks @katokpara

changed the description

changed milestone to %11.10

changed the description

Update Pt 2: The test is currently collecting responses. I'm hoping we can send out another round of social media posts, in order to get more responses before analyzing the results.

I aim to have this issue concluded by the end of this week or early next week.

marked the checklist item 2nd round of advertising on social media to gather more responses [Deadline: Fri Mar 15th] as completed

My apologies for the delay on this issue! Here are the results:

This design evaluation was split into 4 variants of the same test questions with different icons presented in a different order. Test A received 45 responses, Test B and C received 46 responses each, Test D received 48 responses.

Key Findings:

• 51% of users selected issue + checkmark as best representing a closed issue

  • The users who selected this icon stated that they chose it because a checkmark means “done” and communicates that the work for the issue is completed.
  • Some users made the distinction that this icon looks “Positively closed,” which would mean that the issue was resolved. On the other hand, the message may not match when an issue is closed without a resolution.
  • Other users stated that they chose this one because the others looked similar to traffic signs or that they were communicating a more negative meaning for "closed” such as blocked, do not comment, declined, etc.

• When asked to determine the status of an issue

  • Users who saw an issue + checkmark icon (Test A) stated that the issue was Complete (49%) or Open (31%). 0% of users stated that it was Closed. Later on in the test, the majority of respondents selected issue + checkmark as best representing closed, compared to the other icons. Their explanations for why they chose the icon was that it seemed to say the issue is "Done" which supports the fact that "Complete" was the top choice in this question.
  • Users who saw a “ban” icon (Test B) stated that the issue was Blocked (39%), Closed (17%), or that they didn’t know (30%).
  • Users who saw the “Do not enter” icon (Test C) stated that the issue was Open (28%), Merged (13%), Closed (11%), or that they didn’t know (30%). Answers were very split in this test.
  • Users who saw a filled circle icon (Test D) stated that the issue was Open (33%), Complete (23%) or that they didn’t know (31%).

• The majority of users mentioned a distinction between a closed/non-actionable issue and a closed/complete issue. Some say that a closed issue means the task is completed while others say that it is no longer needing attention. Others acknowledge different interpretations of a closed issue including no further action required, resolved, no longer need attention, and declined.

• 78% of users understand the meaning of the blue “merged” icon.

• 54% of users voted that the current red “closed” MR icon represents a “blocked” MR. The second most common answer was "closed" (34%).

• 74% of users understand that the open green circle icon represents an “open” issue.

marked the checklist item Review results and share! [Deadline: Weds Mar 20th] as completed

made the issue visible to everyone

Closing since done.

closed

added Done label and removed In Progress label

removed UX research label

added researched [DEPRECATED] label

mentioned in issue #155 (closed)

mentioned in epic &1286 (closed)

added devopsplan label

mentioned in issue #322 (closed)

mentioned in issue #323 (closed)

mentioned in issue #324 (closed)

mentioned in epic &1733

mentioned in issue #343 (closed)

mentioned in epic &1959 (closed)

mentioned in issue gitlab-design#766 (closed)