Commit c9f89074 authored by Jannik Lehmann's avatar Jannik Lehmann Committed by Jafar Soltani
Browse files

Sentry Signal Loop: measure noise reduction excluding archived issues

parent c644c090
Loading
Loading
Loading
Loading
+16 −2
Changes for content/handbook/company/working-groups/task-groups/sentry-signal-loop.md: 16 added lines, 2 removed lines.
Original line number Diff line number Diff line
@@ -29,7 +29,7 @@ The [Frontend Observability Working Group](/handbook/company/working-groups/fron

The work splits into two phases that together form the loop:

1. **Primary Goal — Reduce noise at the source.** Update the Sentry SDK configuration and Sentry inbound filters so that the project reports unexpected failures of our code, and nothing more. The exit criterion is a ≥50% drop in 30-day event volume from the ~26M baseline (see [Exit Criteria](#exit-criteria) for the cited measurement). The exact filters are decided iteratively as MRs land and Sentry data updates.
1. **Primary Goal — Reduce noise at the source.** Update the Sentry SDK configuration and Sentry inbound filters so that the project reports unexpected failures of our code, and nothing more. The exit criterion is a ≥50% drop in 30-day event volume, excluding events from issues triaged and archived on the platform, from the ~26M baseline (see [Exit Criteria](#exit-criteria) for the cited measurement and the rationale for excluding archived issues). The exact filters are decided iteratively as MRs land and Sentry data updates.

2. **Secondary Goal — Build a Sentry → GitLab issue → Duo Developer triage automation, and run it.** Once the noise floor is lowered, we build the system that keeps us on top of the events. The automation runs on a daily schedule, fetches the latest Sentry issues, opens corresponding GitLab issues routed to the owning group through `feature_category`, and uses Duo Developer to produce triage on each. The unifying pattern: an agent separates noise from signal, and on each side produces a proposal that a human acts on. No MRs are opened autonomously — the handoff to the human is part of the design, not a limitation of it.

@@ -43,9 +43,23 @@ The work splits into two phases that together form the loop:

These are the known challenges, but there may be additional issues that have yet to be identified.

### Measuring noise: why archived-issue events are excluded

Sentry's event-volume statistic counts every accepted event, including those from issues already triaged and archived-until-escalation. The raw number therefore understates the triage-experience improvement.

Suppressing these events at the source doesn't work for shared error handlers: they carry both noise and signal, and removing the capture call deletes both. Archiving in Sentry keeps the signal, resurfaces issues if they escalate, and is reversible. Source-level filters remain the right tool for events that are never signal.

The measurement is the 30-day `count()` via the Sentry events API minus the same query filtered to `is:ignored`. `is:ignored` reflects current issue state, so trailing events of freshly archived issues count retroactively — intended, since the metric reflects the current triage state.

#### Reporting the archived total

Excluding archived events makes the metric sensitive to triage decisions, not just shipped code. An issue archived-until-escalation only resurfaces if it exceeds its own volume forecast, so a steady high-volume issue can stay archived indefinitely.

To keep this visible, the Task Group's closing report states all three figures: raw accepted volume, archived-issue events, and the net. The Task Group establishes the baseline; keeping noise low afterwards is the owning teams' responsibility.

## Exit Criteria

1. **Noise reduced at source (primary outcome).** Sentry SDK configuration and inbound filters are updated such that 30-day event volume in `gitlabcom-clientside` is reduced by ≥50% to less then 12.4M events in total from the March 2026 baseline of ~24.8M error events (see [Context](#context) for the screenshot and live view).
1. **Noise reduced at source and through triage (primary outcome).** Sentry SDK configuration and inbound filters are updated, and known non-actionable issues are archived in Sentry, such that 30-day event volume in `gitlabcom-clientside`, **excluding events from archived issues**, is reduced by ≥50% to less than 12.4M events from the March 2026 baseline of ~24.8M error events (see [Context](#context) for the screenshot and live view, and [Measuring noise](#measuring-noise-why-archived-issue-events-are-excluded) for why archived-issue events are excluded).
2. **Automation exists and has run on real issues.** The automation runs on a daily schedule, fetches Sentry issues, opens corresponding GitLab issues routed through `feature_category`, and produces Duo Developer triage (root cause, code references, proposed fix) on each. The automation has run for at least 2 weeks against real owning groups, with results recorded somewhere reviewable (issue link, Duo's output, owning-group action taken).
3. **The automation is useful enough to keep running.** The main thing we look at: when the automation opens a GitLab issue, does the owning group actually do something with it within 14 days — pick it up, assign it, put it on a milestone, fix it — or do they just close it and move on? That tells us whether Duo's triage is helping or just adding noise. Two extra checks back this up: we read through a handful of Duo's analyses by hand to see if they actually point at the real bug, and we look at how many issues got opened in total (a few good ones is a very different result from a flood of mediocre ones). At the end, the Task Group writes down a clear yes-or-no call with the reasoning. **"No, this didn't work, and here's what we learned" is a fine answer** — we don't pick a target percentage up front, because figuring out what "useful" looks like is the whole point of trying this.
4. **Long-term ownership is resolved.** Either a long-term owner has been identified for the automation, has been involved in the last 4 weeks of the Task Group, and has accepted a documented handover **or** no owner has been found and the automation is sunset at Task Group end with the rationale documented.