<!--IssueSummary start--> <details> <summary> Everyone can contribute. [Help move this issue forward](https://handbook.gitlab.com/handbook/marketing/developer-relations/contributor-success/community-contributors-workflows/#contributor-links) while earning points, leveling up and collecting rewards. </summary> - [Label this issue](https://contributors.gitlab.com/manage-issue?action=label&projectId=69663206&issueIid=48) </details> <!--IssueSummary end--> ## Problem Threshold alert notifications (email and the GitLab issue created from an alert) carry very little context. The body states that an alert fired and that a metric crossed a threshold. It does not show what triggered it. A responder has to click through to the logs or traces view to understand anything, and the link itself is a raw, very long URL. Example, from a real alert issue: ``` description = This alert is fired when the defined metric (current value: 1) crosses the threshold (0) summary = The rule threshold is set to 0, and the observed metric value is 1 related_logs = https://...2000+ characters of double-encoded query JSON... ``` At first glance the reader learns only that a threshold was crossed. Not what was being watched, not what actually matched, and the one link present is unreadable. ## What would actually help, in priority order 1. **A sample of the matching record(s) that triggered the alert.** This is the most valuable addition. For an error-log alert, the actual log line (or a short sample of lines) tells the responder what happened, not just that something happened. It will not always be the full story, but it gives a first-sight cause. 2. **The filter expression that defines the check**, for example `service.name='contributors-svc' AND severity_text='ERROR' AND body NOT CONTAINS 'OpenTelemetry'`. Useful context for what was being watched. 3. **Threshold versus observed value**, presented clearly (already partially present in `summary`). 4. **A friendly, readable link** instead of the raw 2000-character URL. Render it as a Markdown link such as `[View in logs explorer](url)`. A tokenized, labeled layout would read far better than the current prose sentence. For example: ``` **Service:** contributors-svc **Condition:** severity_text = 'ERROR' AND body NOT CONTAINS 'OpenTelemetry' **Threshold:** 0 **Observed:** 1 **Sample log:** <matching log line> [View in logs explorer](https://...) ``` ## Feasibility (verified against the code) All of this is reachable from the threshold rule evaluation path in `pkg/query-service/rules/threshold_rule.go`. - **The rule already holds a `querier.Querier`** (`ThresholdRule.querier`) and already uses it to run the alert's aggregate query (`r.querier.QueryRange(...)` in `buildAndRunQuery`). - **The querier supports raw row queries** (`RequestTypeRaw`), not just aggregates (`RequestTypeTimeSeries`). So a small limited query can fetch the actual matching records. - **The matching-records filter is already built** for the `related_logs` link (`prepareLinksToLogs`). The same condition can drive a sample-record query. - **There is precedent for enrichment.** The exception notification channel (`pkg/alertmanager/alertmanagernotify/gitlab/enricher.go`) already fetches records, details, and traces via a reader to build a rich body. The threshold path can follow the same idea using the querier it already has. The alert template data is currently the limiting factor. `AlertTemplateData` in `pkg/types/ruletypes/templates.go` exposes only `Labels`, `Value`, and `Threshold`. To surface the condition and a sample record, the template data (or the notification body builder) needs those added. ## Proposed direction 1. Fetch a small, capped sample of matching records at alert evaluation, using the existing querier and the same filter used for `related_logs`. Keep the sample size small. 2. Expose the condition expression and the sample record(s) to the alert body, either via `AlertTemplateData` or directly in the notification body builder. 3. Restructure the default alert body into a tokenized, labeled layout: service, condition, threshold versus observed, sample record, and a readable link. 4. Add a readable Markdown link alongside the raw `related_logs` and `related_traces` URLs. This piece is small and independently shippable, it does not depend on the record-fetch work. ## Tradeoff to weigh Fetching a sample record adds one extra query per alert evaluation. This is a cost and design decision for the team: the sample size cap, whether it runs always or only for log and trace alert types, and whether it is opt-in per rule. Flagging it so it is considered rather than assumed. ## Out of scope The exception notification channel (`pkg/alertmanager/alertmanagernotify/gitlab/formatter.go`) already produces a rich body with stacktrace and trace link for exception alerts. That richness depends on exception enrichment data. This issue is about bringing comparable first-sight context to ordinary threshold alerts, not reusing the exception formatter as is. ## Notes - Related: the deep-link prefix fix in [gitlab_o11y!133](https://gitlab.com/gitlab-org/embody-team/experimental-observability/gitlab_o11y/-/merge_requests/133) touches the same threshold alert path, and item 4 here (readable links) is adjacent to it. Surfaced alongside the alert in [contributors-gitlab-com#552](https://gitlab.com/gitlab-org/developer-relations/contributor-success/contributors-gitlab-com/-/issues/552).