Global Search error budget: 98.6% of spend is apdex, and most of it is miscalibrated measurement

Summary

Global Search is 15.5x over its 28d error budget (99.226% availability against a 99.95% SLO — 312 minutes spent against 20.2 minutes allowed). This is not incident-driven: daily availability has sat between 98.3% and 99.6% every single day for 28 days with no outage.

Errors are not the problem. 98.6% of the budget spend is apdex (latency) failure. Every error-kind SLI in the group is at or under 0.03%.

Three components account for 98.5% of the spend, and at least two of them are measurement problems rather than user-facing regressions.

Evidence

All figures pulled from the Global Search error budget dashboard panel queries, run directly against Mimir - Gitlab Gprd on 2026-09-07. The availability figure below reproduces the dashboard's own Availability stat panel exactly (99.2262%), so the attribution below is the dashboard's arithmetic, not a re-derivation.

Budget spend attribution (gprd, 28d)

Component Kind Weight Failures Fail % Share of spend
zoekt_tasks apdex 732,042 75,535 10.32% 44.5%
rails_request apdex 3,634,988 52,116 1.43% 30.7%
global_search apdex 2,088,976 39,509 1.89% 23.3%
sidekiq_execution error 3,908,724 1,343 0.03% 0.8%
rails_request error 3,636,063 961 0.03% 0.6%
global_search_indexing apdex 1,491,181 137 0.01% 0.1%
sidekiq_execution apdex 3,907,449 110 0.00% 0.1%
zoekt_tasks error 699,733 14 0.00% 0.0%
zoekt_searching error 338,749 0 0.00% 0.0%

Daily availability, 28d — flat, not spiky

08-10 99.258%   08-17 98.780%   08-24 99.646%   08-31 98.931%
08-11 99.215%   08-18 98.823%   08-25 98.857%   09-01 97.371%
08-12 98.884%   08-19 98.520%   08-26 99.206%   09-02 99.150%
08-13 98.789%   08-20 98.788%   08-27 99.050%   09-03 99.076%
08-14 98.310%   08-21 98.791%   08-28 99.238%   09-04 98.860%
08-15 98.805%   08-22 98.982%   08-29 99.326%   09-05 99.205%
08-16 98.878%   08-23 98.952%   08-30 99.286%   09-06 99.244%
                                                09-07 99.180%

There is no incident to attribute this to. The SLO is being missed structurally.

Finding 1 — rails_request re-judges search requests against a 5s bar the team already rejected (30.7% of spend)

Search API endpoints declare urgency :low in lib/api/search.rb:39, which resolves to a 5 second apdex target (lib/gitlab/endpoint_attributes/config.rb, RequestUrgency.new(:low, 5)).

The same requests are separately measured by Gitlab::Metrics::GlobalSearchSlis, whose targets are deliberately much higher (lib/gitlab/metrics/global_search_slis.rb:14-20):

Constant Target
ADVANCED_CONTENT_TARGET_S 2.452s
ADVANCED_CODE_TARGET_S 15.52s
ZOEKT_TARGET_S 15.52s
BASIC_CODE_TARGET_S 27.538s
BASIC_MR_TARGET_S 15s

global_search_slis.rb:96-107 enumerates the endpoints it covers — GET /api/:version/search, the project and group search endpoints, SearchController#show and #count. Those are the same endpoints rails_request covers. One user request is scored twice, against two different targets, and both scores land in the same budget.

A 10-second code search passes the search team's considered 15.52s target and simultaneously fails the generic 5s bar. Observed effect at the endpoint level over 6h:

Endpoint Requests/6h Apdex failures Fail %
GET /api/:version/search 581,311 30,124 5.18%
GET /api/:version/projects/:id/(-/)search 413,886 4,683 1.13%
GET /api/:version/groups/:id/(-/)search 511,533 3,266 0.64%
SearchController#autocomplete 134,912 3,238 2.40%
SearchController#count 103,533 2,716 2.62%

The dedicated global_search SLI scores the same traffic at 2.09% over the same window.

graphql_query is already excluded from every stage group's budget via gitlab:ignored_component:stage_group, so component-level exclusion is an established mechanism here, not a novel request.

Open decision — this needs a maintainer opinion, both options are defensible:

  • (a) Exclude rails_request apdex for search endpoints from the group budget, leaving global_search as the single source of truth for search latency. Smaller and reversible. Risk: search endpoints stop contributing to the generic Rails latency view.
  • (b) Reconcile the two — raise urgency on the search endpoints to match the search SLI targets. Keeps one number. Risk: urgency is a coarse enum (0.25 / 0.5 / 1 / 5s) with no value above 5s, so this cannot express a 15.52s target without extending the enum.

Option (b) is not currently expressible. That is an argument for (a), but the choice of which signal is authoritative belongs to the group.

Finding 2 — the global_search SLI targets are calibrated against 2022 data (23.3% of spend)

global_search_slis.rb:11-13 states the targets are "the 99.95th percentile of code searches gathered on 25-10-2022". They have not been re-derived in nearly four years, across the introduction of Zoekt and multiple index changes.

Failure rate by scope over 6h, gprd:

Search type Scope Requests/6h Fail % Target applied
advanced users 442,831 3.91% 2.452s
advanced projects 113,142 5.49% 2.452s
advanced work_items 13,352 5.90% 2.452s
advanced snippet_titles 13,745 5.87% 2.452s
advanced merge_requests 20,902 4.65% 2.452s
advanced notes 30,223 3.36% 2.452s
advanced milestones 69,231 2.56% 2.452s
advanced commits 67,187 1.95% 2.452s
zoekt blobs 281,938 0.05% 15.52s
advanced blobs 42,856 0.08% 15.52s
basic blobs 107,212 0.00% 27.538s

Code search — the scope the targets were actually measured from — passes comfortably. Every failure is concentrated in the non-blob scopes, which all share a single 2.452s target derived from a code search percentile measurement. users alone is 443k requests per 6h, the highest-volume scope in the product, judged against a target nobody measured for it.

duration_target (global_search_slis.rb:66-82) has exactly one branch for all non-code advanced searches. There is no per-scope calibration.

Finding 3 — zoekt_tasks apdex measures queue depth, not indexing latency (44.5% of spend)

The single largest contributor. Already partly understood by the group, and the history matters:

  • Threshold was raised 1800s → 3600s after this analysis showed ~12% of tasks breaching purely from queue delay.
  • Raised again 3600s → 7200s on 2026-08-28 (commit d679e10e6191, "Increase Zoekt tasks SLI apdex threshold to 2 hours"). ee/lib/gitlab/metrics/zoekt_tasks_slis.rb:11.

Measured effect of that doubling, daily zoekt_tasks apdex success:

pre-change  08-21 85.92%  08-24 88.01%  08-27 87.19%
post-change 08-28 93.40%  08-31 95.52%  09-04 88.21%  09-07 92.99%

Doubling the threshold bought roughly 6 percentage points and still leaves ~7% of tasks failing. The 28d aggregate of 10.32% spans both regimes.

The reason is stated plainly in the group's own investigation: the SLI measures the gap between perform_at and updated_at, so it captures queue wait, not processing time. Independently reported in that same issue: ~97% of tasks index in under 1 second. A metric where 97% of the work completes in 1s but 7% of the measurements fail a 2-hour bar is not measuring latency — it is measuring backlog, and raising the threshold a third time will not change that.

Recommendation: stop moving the threshold. Split the measurement into queue wait and processing time, and decide which one is the SLI. Throughput/backlog belongs in a saturation signal, not an apdex.

Why this is worth doing now

Roughly 54% of the budget spend (Findings 1 and 2) comes from measurement choices that were reasonable when made and have not been revisited. A further 44.5% (Finding 3) is a metric measuring something other than what its name claims. Until these are corrected, the group's error budget cannot distinguish a real regression from calibration drift — which is exactly what happened in the 2026-06 investigation, where the initial reading was "all graphs are green."

Correcting the measurement is not the same as declaring the latency acceptable. Findings 1 and 2 may well reveal genuine slowness in the non-blob scopes once the right target is applied. The point is that today nobody can tell.

Proposed work items

  1. Decide the authoritative search latency signal and implement it — exclude rails_request for search endpoints, or extend urgency and reconcile. Open decision above.
  2. Re-derive the GlobalSearchSlis targets from current production data, per scope rather than one target for all non-code searches. Document the measurement date and source in the constant comments as the 2022 version did.
  3. Split zoekt_tasks queue wait from processing time; make the apdex measure processing and move backlog to a saturation/throughput signal.

Done when

  • The group's 28d error budget spend can be attributed to a specific, defensible cause per component, with no component contributing >5% of spend from a threshold nobody can justify from current data.
  • Each of the three SLI changes above is either merged or explicitly rejected with the reasoning recorded in this issue.

Not in scope

Origin

Follow-up to https://gitlab.com/gitlab-org/gitlab/-/work_items/604150 (Investigate Error Budget Rates for Global Search 2026-06), which established the zoekt queue-delay cause but did not examine the double-counting or the target calibration. Monthly report: error-budget-reports#73.

Queries and raw output are reproducible via the Grafana datasource proxy against mimir-gitlab-gprd; the panel expressions used are the dashboard's own, unmodified apart from template variable substitution.