Commit bb2b90d1 authored by Jannik Lehmann's avatar Jannik Lehmann 🚀
Browse files

Close the Sentry Signal Loop Task Group with a closing report

parent 75d2672f
Loading
Loading
Loading
Loading
+1 −1
Changes for content/handbook/company/working-groups/task-groups/_index.md: 1 added line, 1 removed line.
Original line number Diff line number Diff line
@@ -101,11 +101,11 @@ task forward.
1. [Accessibility Audit Triage](https://internal.gitlab.com/handbook/company/internal-working-groups/internal-task-groups/accessibility-audit-triage)
1. [BootstrapVue Removal](bootstrap-vue-removal/)
1. [Intelligent Model Selection](intelligent-model-selection/)
1. [Sentry Signal Loop](sentry-signal-loop/)

## Past Task Groups (alphabetic order)

1. [CSS Utilities](css-utils/)
1. [GitLab UI @vue/compat Compatibility](gitlab-ui-vue-compat/)
1. [Sentry Signal Loop](sentry-signal-loop/)
1. [Vue 3 Quarantined Tests](vue3-quarantined-tests/)
1. [Vue 3 Router 4 Test Compatibility](vue3-router4-tests/)
+71 −1
Changes for content/handbook/company/working-groups/task-groups/sentry-signal-loop.md: 71 added lines, 1 removed line.
Original line number Diff line number Diff line
@@ -9,6 +9,10 @@ description: "A Task Group restoring frontend Sentry as a useful triage tool: cu
| --------------- | -------------------------------------------------------------------------------------------------------------- |
| Date Created    | 2026-05-19                                                                                               |
| Target End Date | 2026-09-27                                                                                                     |
| End Date        | 2026-09-28                                                                                                     |
| Current Status  | Closed. The automation keeps running in maintenance mode (see [Closing report](#closing-report))              |
| Project         | [duo-sentry-insights](https://gitlab.com/gitlab-org/frontend/duo-sentry-insights)                              |
| Epic            | [gitlab-org&22089](https://gitlab.com/groups/gitlab-org/-/epics/22089)                                        |
| Slack           | [#tg_sentry_signal_loop](https://gitlab.enterprise.slack.com/archives/C0B41EVB4J3) (only accessible from within the company)       |

## Context
@@ -64,8 +68,74 @@ To keep this visible, the Task Group's closing report states all three figures:
3. **The automation is useful enough to keep running.** The main thing we look at: when the automation opens a GitLab issue, does the owning group actually do something with it within 14 days — pick it up, assign it, put it on a milestone, fix it — or do they just close it and move on? That tells us whether Duo's triage is helping or just adding noise. Two extra checks back this up: we read through a handful of Duo's analyses by hand to see if they actually point at the real bug, and we look at how many issues got opened in total (a few good ones is a very different result from a flood of mediocre ones). At the end, the Task Group writes down a clear yes-or-no call with the reasoning. **"No, this didn't work, and here's what we learned" is a fine answer** — we don't pick a target percentage up front, because figuring out what "useful" looks like is the whole point of trying this.
4. **Long-term ownership is resolved.** Either a long-term owner has been identified for the automation, has been involved in the last 4 weeks of the Task Group, and has accepted a documented handover **or** no owner has been found and the automation is sunset at Task Group end with the rationale documented.

## Closing report

The Task Group closed on 2026-09-28. Exit criteria 1 to 3 are met. Exit criterion 4 is resolved with an amendment: no group took over long-term ownership of the automation, and it was not sunset either. It continues in maintenance mode under the former DRI. The reasoning is below.

### 1. Noise reduction

All three figures, measured on 2026-09-28 over the trailing 30 days through the Sentry events API (`project=4`, `dataset=errors`, `statsPeriod=30d`, `field=count()`):

| Figure                                             | Events         |
| -------------------------------------------------- | -------------- |
| Raw accepted volume                                | 7,903,384      |
| Events from archived issues (`is:ignored`)         | 4,558,842      |
| **Net (raw minus archived)**                       | **3,344,542**  |
| March 2026 baseline                                | ~24,800,000    |
| Target (at least 50% reduction)                    | <12,400,000    |

The net figure is down 86.5% from the baseline; the raw figure alone is down 68%. Both are well inside the target.

![Sentry Discover, gitlabcom-clientside, September 2026: 7.9M error events in 30 days](/images/company/working-groups/task-groups/sentry-signal-loop/closing-september-2026.png)

Closing measurement: 7.9M raw error events in the 30 days to 2026-09-28, compared to ~24.8M in March.

Where the reduction came from: source-level filters in the Sentry SDK configuration (non-actionable error patterns, expected `4xx` responses, extension noise), Sentry inbound filters, and archiving known non-actionable issues in Sentry so they no longer surface in triage.

### 2. The automation

The Sentry Signal Loop is a scheduled pipeline in [duo-sentry-insights](https://gitlab.com/gitlab-org/frontend/duo-sentry-insights). Once a day (or week, per schedule) it fetches the new Sentry issues for one `feature_category`, mirrors each into a confidential `gitlab-org/gitlab` issue, and asks Duo Developer to triage it. Humans decide; the pipeline does the legwork.

What shipped:

- Fetch, enrich, reconcile, plan, apply stages. Dry-run by default, with a human-readable plan as a pipeline artifact.
- Deduplication through a `<!-- sentry-issue-id -->` marker in the issue description, plus a link back from the Sentry issue.
- Routing by `feature_category`, with DRI assignment, optional group label, and optional parent epic.
- Duo Developer triage on every created or reopened issue: Signal or Noise verdict, confidence, evidence level, root cause, code references, proposed fix, and a recommended Sentry disposition.
- A dominance gate: an error is only mirrored for the category that produces most of its events, so shared components (navigation, session modal, timestamps) do not land in every team's run.
- A per-run creation cap ranked by affected users, added after a dry run for a high-volume category showed it would have opened about 100 issues.
- Release-aware regression detection: a closed issue is only reopened when a release created after the close throws the error. Events from long-lived browser tabs on old bundles get a single comment instead of a reopen.
- Onboarding is one pipeline schedule; offboarding is turning it off.

The loop ran from 2026-05-22 to Task Group close and mirrored 258 Sentry issues to 9 groups. Six groups have set up a pipeline schedule for their feature category; five of those schedules are active at close.

The original plan scaled from opt-in to default opt-out. We stayed opt-in. Opening issues for teams that did not ask for them was the biggest risk to trust in the automation, and slower adoption was the better trade.

### 3. Is it useful enough to keep running? Yes

Measured on 2026-09-28 across all issues labeled `Task Group::Sentry Signal Loop`:

- **78% pickup.** Of the 154 mirrored issues older than 14 days, 120 were assigned, put on a milestone, closed, or linked to a merge request by the owning group. 50 were closed, 24 of them through a fix MR. Four groups acted on every issue they received.
- **Duo triage on 248 of 258 issues.** 151 Signal, 97 Noise; 193 backed by a resolved stack trace. Signal verdicts led to 44 fix MRs, Noise verdicts to 2. Owning groups treat the verdict as a usable first pass.
- **No flood, no duplicates.** No group asked to stop, and the pipeline never opened two GitLab issues for the same Sentry issue.

### 4. Ownership: maintenance mode under the former DRI

Exit criterion 4 allowed two outcomes: an owning group, or sunset. Neither happened.

Developer Experience and the Observability team were both asked and both declined for capacity reasons. They confirmed Sentry itself is in maintenance mode and has no usage owner, and that self-service ownership ("you build it, you run it") is the expected default going forward.

Sunsetting a tool that five groups actively rely on, and that they were encouraged to adopt, would have been the worse outcome. So the resolution is:

- The automation stays running. Each consuming group owns its own pipeline schedule and can switch it off at any time.
- The former DRI maintains the shared repository, the two access tokens it runs on, and their rotation. Scope is bug fixes and breakage caused by Sentry or GitLab API changes. No new features. Expected effort is close to zero hours per month.
- Operational details live in the project's `OPERATIONS.md`.
- Kill switch: disable the pipeline schedules. Nothing else runs.
- This arrangement is re-evaluated if the maintainer changes role, if token custody cannot be kept, or if a group wants to take ownership.

## Roles and Responsibilities

| Task Group Role            | Person          | Title                                                        |
| --------------- | --------------- | ------------------------------------------------------------ |
| -------------------------- | --------------- | ------------------------------------------------------------ |
| DRI                        | [Jannik Lehmann](https://gitlab.com/jannik_lehmann)  | Senior Frontend Engineer, AI-Powered:AI Catalog              |
| Maintainer (after close)   | [Jannik Lehmann](https://gitlab.com/jannik_lehmann)  | Senior Frontend Engineer, AI-Powered:AI Catalog              |