Global Search apdex: the 4-hourly degradation burst doubled in depth and count on 2026-09-09
global_search apdex on gprd regressed at 2026-09-09 15:35Z: degradation bursts roughly doubled in count and depth, and 28d availability is now 99.052% against a 99.95% SLO, 18.96x over budget. No incident is currently open for this. Apdex recovers to 99.0-99.5% between bursts, so nothing is continuously broken — the whole budget loss is in the bursts.
That pattern fits a periodic Elasticsearch-cluster-side maintenance or batch operation that doubled in scope and gained a second 4-hourly cycle, though nothing measured so far confirms it.
Could someone with view access to the gprd Advanced Search Elasticsearch cluster check what periodic operation runs there every ~4 hours, and what changed about it at 2026-09-09 15:35Z — whether it gained a second schedule or doubled its work?
References
Causes ruled out with their evidence — Rails build, feature flags, indexing workers, SLI composition, traffic growth: $6054134
🤖 Onset timestamps, discriminating measurements and gaps (for deep readers)
Mechanism and citations
All figures from Mimir gprd, read-only, 5m resolution. Apdex here has always been sawtoothed, with one burst every ~4h, metronomic from 09-06 until 2026-09-09 12:30Z. The global_search component then leaves its 99.7-100% plateau at 14:53Z for a 20m dip; the first oversized burst starts 15:35Z and bottoms at 64.8% apdex at 15:57Z. Bursts went from 6/day at 35-45 min long and 23-46% peak fail on search_type="advanced" to 12/day at 65-85 min and 63-83%, the new ones starting around :20 past the hour, first seen 09-10 02:20Z. Daily apdex: 09-08 98.526, 09-09 98.461, 09-10 96.268, 09-11 96.059.
What separates a cluster-side cause from a Rails one: search_type="advanced" went 0.6% -> 16-63% fail while zoekt and basic stayed at 0.02%; blobs, served by Zoekt, fail at 0.08% while Elasticsearch-backed scopes run 29.5-47.4%; fail rates sit within 0.3pp across us-east1-b/c/d and across main 34.2% vs cny 34.3%, and cny runs a different, newer Rails build; gstg holds 100.0% apdex throughout. search_level="global" went 10.0% -> 31.0% against project 1.4% -> 3.3%, so unfiltered cross-index queries hurt worst, and GET /api/:version/search errors went 0.000 -> 0.329 rps, consistent with cluster timeouts surfacing as 5xx.
What I did not verify
Anything on the cluster itself. No thread-pool queue or rejection, shard-merge, snapshot or force-merge metric for the Advanced Search cluster appears in gprd Mimir, which carries elasticsearch_* series only for the logging cluster.
@johnmason to leave feedback.