## Summary Global Search is **15.5x over its 28d error budget** (99.226% availability against a 99.95% SLO — 312 minutes spent against 20.2 minutes allowed). This is not incident-driven: daily availability has sat between 98.3% and 99.6% every single day for 28 days with no outage. **Errors are not the problem.** 98.6% of the budget spend is apdex (latency) failure. Every error-kind SLI in the group is at or under 0.03%. Three components account for **98.5%** of the spend, and at least two of them are measurement problems rather than user-facing regressions. ## Evidence All figures pulled from the [Global Search error budget dashboard](https://dashboards.gitlab.net/d/stage-groups-detail-global_search/) panel queries, run directly against `Mimir - Gitlab Gprd` on 2026-09-07. The availability figure below reproduces the dashboard's own `Availability` stat panel exactly (99.2262%), so the attribution below is the dashboard's arithmetic, not a re-derivation. ### Budget spend attribution (gprd, 28d) | Component | Kind | Weight | Failures | Fail % | Share of spend | |---|---|---|---|---|---| | `zoekt_tasks` | apdex | 732,042 | 75,535 | 10.32% | **44.5%** | | `rails_request` | apdex | 3,634,988 | 52,116 | 1.43% | **30.7%** | | `global_search` | apdex | 2,088,976 | 39,509 | 1.89% | **23.3%** | | `sidekiq_execution` | error | 3,908,724 | 1,343 | 0.03% | 0.8% | | `rails_request` | error | 3,636,063 | 961 | 0.03% | 0.6% | | `global_search_indexing` | apdex | 1,491,181 | 137 | 0.01% | 0.1% | | `sidekiq_execution` | apdex | 3,907,449 | 110 | 0.00% | 0.1% | | `zoekt_tasks` | error | 699,733 | 14 | 0.00% | 0.0% | | `zoekt_searching` | error | 338,749 | 0 | 0.00% | 0.0% | ### Daily availability, 28d — flat, not spiky ``` 08-10 99.258% 08-17 98.780% 08-24 99.646% 08-31 98.931% 08-11 99.215% 08-18 98.823% 08-25 98.857% 09-01 97.371% 08-12 98.884% 08-19 98.520% 08-26 99.206% 09-02 99.150% 08-13 98.789% 08-20 98.788% 08-27 99.050% 09-03 99.076% 08-14 98.310% 08-21 98.791% 08-28 99.238% 09-04 98.860% 08-15 98.805% 08-22 98.982% 08-29 99.326% 09-05 99.205% 08-16 98.878% 08-23 98.952% 08-30 99.286% 09-06 99.244% 09-07 99.180% ``` There is no incident to attribute this to. The SLO is being missed structurally. ## Finding 1 — `rails_request` re-judges search requests against a 5s bar the team already rejected (30.7% of spend) Search API endpoints declare `urgency :low` in `lib/api/search.rb:39`, which resolves to a **5 second** apdex target (`lib/gitlab/endpoint_attributes/config.rb`, `RequestUrgency.new(:low, 5)`). The same requests are separately measured by `Gitlab::Metrics::GlobalSearchSlis`, whose targets are deliberately much higher (`lib/gitlab/metrics/global_search_slis.rb:14-20`): | Constant | Target | |---|---| | `ADVANCED_CONTENT_TARGET_S` | 2.452s | | `ADVANCED_CODE_TARGET_S` | 15.52s | | `ZOEKT_TARGET_S` | 15.52s | | `BASIC_CODE_TARGET_S` | 27.538s | | `BASIC_MR_TARGET_S` | 15s | `global_search_slis.rb:96-107` enumerates the endpoints it covers — `GET /api/:version/search`, the project and group search endpoints, `SearchController#show` and `#count`. Those are the same endpoints `rails_request` covers. **One user request is scored twice, against two different targets, and both scores land in the same budget.** A 10-second code search passes the search team's considered 15.52s target and simultaneously fails the generic 5s bar. Observed effect at the endpoint level over 6h: | Endpoint | Requests/6h | Apdex failures | Fail % | |---|---|---|---| | `GET /api/:version/search` | 581,311 | 30,124 | **5.18%** | | `GET /api/:version/projects/:id/(-/)search` | 413,886 | 4,683 | 1.13% | | `GET /api/:version/groups/:id/(-/)search` | 511,533 | 3,266 | 0.64% | | `SearchController#autocomplete` | 134,912 | 3,238 | 2.40% | | `SearchController#count` | 103,533 | 2,716 | 2.62% | The dedicated `global_search` SLI scores the same traffic at 2.09% over the same window. `graphql_query` is already excluded from every stage group's budget via `gitlab:ignored_component:stage_group`, so component-level exclusion is an established mechanism here, not a novel request. **Open decision — this needs a maintainer opinion, both options are defensible:** - **(a) Exclude `rails_request` apdex for search endpoints** from the group budget, leaving `global_search` as the single source of truth for search latency. Smaller and reversible. Risk: search endpoints stop contributing to the generic Rails latency view. - **(b) Reconcile the two** — raise `urgency` on the search endpoints to match the search SLI targets. Keeps one number. Risk: `urgency` is a coarse enum (0.25 / 0.5 / 1 / 5s) with no value above 5s, so this cannot express a 15.52s target without extending the enum. Option (b) is not currently expressible. That is an argument for (a), but the choice of which signal is authoritative belongs to the group. ## Finding 2 — the `global_search` SLI targets are calibrated against 2022 data (23.3% of spend) `global_search_slis.rb:11-13` states the targets are *"the 99.95th percentile of code searches gathered on 25-10-2022"*. They have not been re-derived in nearly four years, across the introduction of Zoekt and multiple index changes. Failure rate by scope over 6h, gprd: | Search type | Scope | Requests/6h | Fail % | Target applied | |---|---|---|---|---| | advanced | users | 442,831 | **3.91%** | 2.452s | | advanced | projects | 113,142 | **5.49%** | 2.452s | | advanced | work_items | 13,352 | **5.90%** | 2.452s | | advanced | snippet_titles | 13,745 | **5.87%** | 2.452s | | advanced | merge_requests | 20,902 | 4.65% | 2.452s | | advanced | notes | 30,223 | 3.36% | 2.452s | | advanced | milestones | 69,231 | 2.56% | 2.452s | | advanced | commits | 67,187 | 1.95% | 2.452s | | zoekt | blobs | 281,938 | 0.05% | 15.52s | | advanced | blobs | 42,856 | 0.08% | 15.52s | | basic | blobs | 107,212 | 0.00% | 27.538s | Code search — the scope the targets were actually measured from — passes comfortably. Every failure is concentrated in the non-blob scopes, which all share a single 2.452s target derived from a *code search* percentile measurement. `users` alone is 443k requests per 6h, the highest-volume scope in the product, judged against a target nobody measured for it. `duration_target` (`global_search_slis.rb:66-82`) has exactly one branch for all non-code advanced searches. There is no per-scope calibration. ## Finding 3 — `zoekt_tasks` apdex measures queue depth, not indexing latency (44.5% of spend) The single largest contributor. Already partly understood by the group, and the history matters: - Threshold was raised 1800s → 3600s after [this analysis](https://gitlab.com/gitlab-org/gitlab/-/work_items/604150#note_3746196334) showed ~12% of tasks breaching purely from queue delay. - Raised again 3600s → **7200s** on 2026-08-28 (commit `d679e10e6191`, *"Increase Zoekt tasks SLI apdex threshold to 2 hours"*). `ee/lib/gitlab/metrics/zoekt_tasks_slis.rb:11`. Measured effect of that doubling, daily `zoekt_tasks` apdex success: ``` pre-change 08-21 85.92% 08-24 88.01% 08-27 87.19% post-change 08-28 93.40% 08-31 95.52% 09-04 88.21% 09-07 92.99% ``` Doubling the threshold bought roughly 6 percentage points and **still leaves ~7% of tasks failing**. The 28d aggregate of 10.32% spans both regimes. The reason is stated plainly in the group's own investigation: the SLI measures the gap between `perform_at` and `updated_at`, so it captures **queue wait**, not processing time. Independently reported in that same issue: ~97% of tasks index in under 1 second. A metric where 97% of the work completes in 1s but 7% of the measurements fail a 2-hour bar is not measuring latency — it is measuring backlog, and raising the threshold a third time will not change that. **Recommendation:** stop moving the threshold. Split the measurement into queue wait and processing time, and decide which one is the SLI. Throughput/backlog belongs in a saturation signal, not an apdex. ## Why this is worth doing now Roughly **54% of the budget spend** (Findings 1 and 2) comes from measurement choices that were reasonable when made and have not been revisited. A further 44.5% (Finding 3) is a metric measuring something other than what its name claims. Until these are corrected, the group's error budget cannot distinguish a real regression from calibration drift — which is exactly what happened in the 2026-06 investigation, where the initial reading was "all graphs are green." Correcting the measurement is not the same as declaring the latency acceptable. Findings 1 and 2 may well reveal genuine slowness in the non-blob scopes once the right target is applied. The point is that today nobody can tell. ## Proposed work items 1. **Decide the authoritative search latency signal** and implement it — exclude `rails_request` for search endpoints, or extend `urgency` and reconcile. Open decision above. 2. **Re-derive the `GlobalSearchSlis` targets from current production data**, per scope rather than one target for all non-code searches. Document the measurement date and source in the constant comments as the 2022 version did. 3. **Split `zoekt_tasks` queue wait from processing time**; make the apdex measure processing and move backlog to a saturation/throughput signal. ## Done when - The group's 28d error budget spend can be attributed to a specific, defensible cause per component, with no component contributing >5% of spend from a threshold nobody can justify from current data. - Each of the three SLI changes above is either merged or explicitly rejected with the reasoning recorded in this issue. ## Not in scope - Making search faster. This issue is about measuring it correctly first. - Zoekt indexing throughput/concurrency tuning, which is tracked in the infrastructure MRs referenced from https://gitlab.com/gitlab-org/gitlab/-/work_items/604150. ## Origin Follow-up to https://gitlab.com/gitlab-org/gitlab/-/work_items/604150 (Investigate Error Budget Rates for Global Search 2026-06), which established the zoekt queue-delay cause but did not examine the double-counting or the target calibration. Monthly report: https://gitlab.com/gitlab-org/error-budget-reports/-/work_items/73. Queries and raw output are reproducible via the Grafana datasource proxy against `mimir-gitlab-gprd`; the panel expressions used are the dashboard's own, unmodified apart from template variable substitution.