Ignore errors from Sentry in GitLab

Problem to solve

Errors are noisy and plentiful which makes triage processes time-consuming because it is difficult to sort through the cruft to find the critical ones. By providing the ability to ignore an error in the GitLab UI, we give users another tool to clear out errors that don't need attention, allowing them to easily focus on the one's that require fixing. Ignoring non-critical errors makes the error list easy to scan and triage.

Intended users

• Delaney (Development Team Lead)
• Sasha (Software Developer)
• Devon (DevOps Engineer)
• Sidney (Systems Administrator)

Further details

This work supports the Error Tracking Vision.

Original Proposal

Add the option to ignore Sentry errors in GitLab. It should be available in the list view and on the detail page.
There is a great discussion regarding implemetation of the ignore feature in this comment.

Updated Proposal (WIP)

List ignore button	Error detail ignore button

• Ignore will hide the item from the error list.
• Our default "ignore" behavior will follow Sentry's: "ignore indefinitely"
• Ideally, the "ignore" functionality will work in both directions (ie, will account for both the User ignores error in Sentry -> Error automatically shows ignored status in GitLab and User ignores error in GitLab -> Error automatically shows ignored status in Sentry use cases)

Permissions and Security

Documentation

Testing

What does success look like, and how can we measure that?

What is the type of buyer?

Links / references

added typefeature + 1 deleted label

added to epic &2029

changed milestone to %12.7

added Category:Error Tracking devopsmonitor direction documentation grouprespond release post item + 1 deleted label

@ameliabauerly @nadia_sotnikova Here is the issue for ignoring errors scheduled for %12.7

Added to the planning issue

mentioned in issue gitlab-org/monitor/general#48 (closed)

mentioned in issue #33156 (closed)

changed the description

added [deprecated] Accepting merge requests + 1 deleted label

marked this issue as related to #36501 (closed)

mentioned in epic &2200

assigned to @ameliabauerly

removed [deprecated] Accepting merge requests + 1 deleted label

Posting first thoughts for the design for this issue: we created original designs for the ignore functionality as part of #32464 (comment 227656260). When we tested the first design for the list page, we discovered that users found the icon buttons hard to understand (even though, for the most part, the icon buttons exist elsewhere on GitLab). I'm not sure, though, if the difficulty in understanding the buttons was because we were showing them a static view, where they couldn't hover over the icon to get more information, or if the icons we were using weren't quite right.

I don't believe we currently have an "ignore" case in GitLab. The security dashboard uses dismiss, but I'm not sure that's exactly the same scenario. Here's the dismiss button on the security dashboard:

In thinking about this further, ignore in this scenario really means, stop notifying me about it the error. We do have an existing icon for notifications-off. What I'm wondering is: if we utilize the notifications-off icon for "ignore" and add a tooltip to further explain what the icon button means, will that be sufficient to address the users' concerns?

If not, I suppose we could go to a full-text button. The downside of the full-text button option is that I'm guessing we may also want to add a "resolve" button into the list view at some point, as well. If we do so, we suddenly have a lot of valuable real-estate being taken up by action buttons. But, adding in a full-text button would certainly clarify what that button does.

Here are the visuals for both options:

List view with icon	List view with text button	Detail view with ignore button

I'm leaning towards the icon button option because I think, with the change of icon and the addition of the tooltip, it is probably both clearer than it was and is something that's easily learnable (ie, the first time a user hovers over the button, it will no longer be mysterious). But, WDYT, @sarahwaldner and @nadia_sotnikova? Do you have a preference towards one or the other of these versions?

Also - I know Sentry has multiple options hidden under ignore:

I presume we will tackle these sorts of additional options at some point but, I'm also assuming it's not necessary to add them in until subsequent iterations. Do either of you disagree and think they should be part of this first iteration?

Hi @ameliabauerly ! Thanks for working on this! Really great thoughts.

In thinking about this further, ignore in this scenario really means, stop notifying me about it the error. We do have an existing icon for notifications-off. What I'm wondering is: if we utilize the notifications-off icon for "ignore" and add a tooltip to further explain what the icon button means, will that be sufficient to address the users' concerns?

By default, does "Ignore" button in Sentry ignore the error until it occurs again? Or does it permanently quiet it?

Two of the interview participants I spoke with used Rollbar for error tracking and mentioned the "Permanently Ignore" option that they frequently use, however it wasn't the default option. The default "Ignore" action only ignored that particular instance of the error.

We need to make sure that our default "Ignore" action matches the default Ignore action in Sentry.

As far as the icon goes, I think we need to make sure that its meaning accurately maps to the actual action being performed. So, for example, if "Ignore" by default simply hides the error, the icon currently used for dismissing vulnerability could actually work, especially since it's also the icon used in Sentry. I personally like the "eye" icon for "hide", but I think I may have a lot of designer bias, since literally all designer tools use the "eye" button to hide layers.

On the other hand, if the "Ignore" button by default quiets notifications about the future instances of the same error, the little bell icon could be a great choice.

I think we should go for an icon + tooltip in the list, and a text button (or icon + text if we have such buttons anywhere else in GitLab?) in the Error Detailt page. I agree with you that static mock-ups could have contributed to extra confusion around the buttons. In fact, one of the participants said "The buttons are unclear, I'd have to hover over them to know what they do for sure".

I presume we will tackle these sorts of additional options at some point but, I'm also assuming it's not necessary to add them in until subsequent iterations. Do either of you disagree and think they should be part of this first iteration?

I agree, these can be tackled in the future, not as part of this MVC.

Let me know what you think!

@ameliabauerly @nadia_sotnikova

I agree with the icon + hover tooltip! I also agree the icon should specifically relate to the action and I'll let you determine what it should be.

What happens when someone ignores an error?

I believe that we should use the Sentry API to update the Sentry issue's status to ignored. We also have the option to delete an error, but I don't think that we want to do that.

I'd like us to determine what happens when a user uses the API to update the "status" attribute of a Sentry issue (error) to "ignored". I am thinking that we can use this as the MVC.

Once an error has been ignored, I don't think we should remove it from the list, but perhaps grey it out? and maybe a future iteration would be to provide a way to un-ignore?

WDYT?

I agree, we should follow whatever Sentry does. @sarahwaldner

One of the insights from the Error Tracking UX Research was that users expect ignored errors to still be accessible, hidden in a separate Ignored list/ category.

I think we should hide ignored errors from the list to keep the noise to the minimum. Dealing with a lengthy list of potentially important errors in a sea of unimportant stuff is a big obstacle we're trying to minimize. But we should make all ignored errors accessible via the filter dropdown under the Ignored category.

What do you think?

cc @ameliabauerly

Another thing I wanted to note @sarahwaldner and @ameliabauerly , we should also offer a Resolve action in the error detail view (not in the list). As far as I understand, it basically marks the error as "done". The error is hidden from the main list view and placed in the Resolved category.

If the user resolves an error that has an issue linked, perhaps it could close the associated issue?

What do you think?

@nadia_sotnikova @ameliabauerly

Yes - I think that adding a resolved option is a good idea. There is a Sentry API call to resolve a Sentry issue. I have created #37955 (closed) to capture discussion on resolving errors from GitLab.

Here are the two basic journeys for Ignoring errors:

• User ignores error in Sentry -> Error automatically shows ignored status in GitLab
• User ignores error in GitLab -> Error automatically shows ignored status in Sentry

@syasonik Would you be able to help us determine what happens to an error in Sentry when you use the API to change the status to "ignored"? (API docs linked here) I am curious what it looks like in Sentry and for how long it gets ignored.

@sarahwaldner

From https://docs.sentry.io/workflow/notifications/#issue-states:

Regardless of alert rules, new events in ignored issues will not send alerts. Also, by default, ignored events are hidden in your project’s issue stream. Search the issue stream for is:ignored to find them.

How long the issue gets ignored in Sentry is configurable. The issue can be ignored permanently, or based on a particular criteria:

By Time	By Occurrences	By # of Affected Users	Appearance of Ignored Issue

@syasonik - since the default is configurable in Sentry, could we select which one we think makes most sense for us? For instance, could we decide that the error is ignored until it occurs again? Would something like that seem reasonable to you?

@ameliabauerly I think we could certainly choose a default! If we opted for 'ignore until re-occurs', I think that's functionally pretty similar to resolving the issue.

One of the big reasons to use Ignore is if an alert is making a lot of noise. Maybe you've fixed the problem in the next release, but it won't deploy until next Tuesday. Or maybe you're trying to debug, but the alert is firing hundreds of times per minute and you just want it to shut up for an hour while you solve the problem.

With these fairly common scenarios, we're truly "ignoring" the issue for the time being, even though it's still firing. If I was going to pick a default, I'd probably pick a slightly more aggressive one - probably an indefinite ignore.

Okay, thanks!! I added this question to our team call for tomorrow so we can see what everyone thinks :)

@syasonik Is it possible to select the amount of time that someone wants to ignore an error for if they are ignoring it via the API? It did not appear that way to me.

I agree that an indefinite ignore is a good default to start with and someone could also go into Sentry to revive an error.

@sarahwaldner It is possible, though it is unlisted in their API docs. I found the various controls listed in their API code directly. I gave it a test and it works.

I did see this old feature request for Sentry, though, which might point back towards "ignore until next occurrence" - https://github.com/getsentry/sentry/issues/8835. Would definitely be good to get some group perspective.

added workflowdesign + 1 deleted label

marked this issue as related to #37955 (closed)

@sarahwaldner and @nadia_sotnikova - thanks for all the great feedback. I ran a quick straw poll on slack to test the icon button options:

Option 1: icon from security dashboard	Option 2: eye-slash icon	Option 3: mute notifications icon

Results

• Option 1: People thought the icon from the security dashboard meant dismiss (x2), mark as not an error, block or not allowed, abort. Conclusion: it's not a great fit for "ignore."
• Option 2: Everyone unanimously thought it meant "hide". Conclusion: seems like a viable option for "ignore."
• Option 3: Opinions varied. People thought it meant mute (x2), don’t remind me/alert me about this error, remove notifications, don’t give me notifications. Conclusion: while "mute" does seem accurate, the close linking to notifications doesn't seem quite accurate here.
• One person also wondered if people would know this is a button or not. Since we are also using icon buttons as actions on lists in the security dashboard and on the tags page, I'm less concerned about this point. Perhaps if we get additional feedback on this point, we can re-assess later.

Proposal

Based on this feedback, I propose we move forward with the following:

List ignore button	Error detail ignore button

• Ignore will hide the item from the list.
• Our default "ignore" behavior will follow Sentry's (whether it's to ignore for a certain time period or until it occurs again). The final decision here will require engineering investigation.
• Ideally, the "ignore" functionality will work in both directions (ie, will account for both the User ignores error in Sentry -> Error automatically shows ignored status in GitLab and User ignores error in GitLab -> Error automatically shows ignored status in Sentry use cases)

I will add all of these details to the issue description. Is there anything else that I'm missing or that's needed here, from your POV, @sarahwaldner?

@ameliabauerly Designs look great. Thanks for updating the description. We are ready to chat with engineering about it. I've added it to the agenda for the Thursday meeting.

This is great, thank you!

changed the description

mentioned in issue gitlab-org/monitor/general#52 (closed)

changed the description

As discussed during our weekly. We can break this into two tiny iterations

• Add to detail page
• Add to list

We want to tackle the detail page one first and ideally complete both tiny iterations in one milestone so that we don't end up with two feature blocks in two releases that basically implements the same functionality for users

@ClemMakesApps @ameliabauerly @syasonik - I'm checking in on this issue while Sarah is OOO. What remains to ensure it is ready for dev in 12.7?

I believe the "ready for development" label can be applied. I'll update it now :)

added workflowready for development + 1 deleted label and removed workflowdesign + 1 deleted label

promoted to epic &2315