Skip and best-fit Zoekt negative-unclaimed-storage eviction
What does this MR do and why?
NodeWithNegativeUnclaimedStorageEventWorker evicts indices from a Zoekt node whose reservations exceed its usable storage. Two problems:
- It evicts when the disk isn't full. Indices reserve about 1.67× what they use (
STORAGE_IDEAL_PERCENT_USED = 0.6), so negative unclaimed storage often means only that reservations ran ahead. Rollout already places nothing on such a node (available_for_search_indexingrequires positive unclaimed storage). - It picks indices by ID.
find_eachwalks primary keys, so the oldest index goes first whatever its size.
Behind zoekt_negative_storage_eviction_best_fit (gitlab_com_derisk, off by default; off keeps today's behavior):
- Skip a node whose disk usage is under the high watermark (75%), and log it.
- Otherwise evict the smallest
readyindex whosereserved_storage_bytescovers the shortfall. When none does, evict the largest first until it's covered.
Why: the 2026-10-29 eviction on gprd
- At 04:11 UTC the worker found node
gitlab-zoekt-26at −1.67 GB unclaimed storage and evicted one index: the node's lowest ID, 36.6 GB reserved, 22.0 GB used (healthy). That namespace lost exact code search, and its re-rollout failed, sozoekt_rollout_retry_interval(1d) keeps it without for another day. - The node's disk was 58.3% used at 03:54 and 59.3% at 08:22, under the 75% high watermark.
- 155 smaller
readyindices on the node each covered the shortfall on their own; the best fit was 1.68 GB. - The same node was negative at 00:08 UTC (−3.4 GB) too.
Evidence from Kibana and Database Lab clones of production; aggregate figures only. Details: #635711 (comment 4114927787)
References
- Related to #635711 (Zoekt eviction plan; this is MR 3)
- Rollout: #636993
- Node-level eviction safeguards, which don't cover this worker: !264628 (merged)
Screenshots or screen recordings
Not applicable: backend only, no UI change.
Database
Two new queries, both through Search::Zoekt::Index on one node's indices. They run only when the flag is on and the node's disk is over the high watermark, at most hourly per batch of nodes with negative unclaimed storage. The second runs only when the first returns nothing.
Both use index_zoekt_indices_on_state_and_zoekt_node_id (state, zoekt_node_id) and sort the node's ready indices: at most 12,825 per node on gprd. No new index.
1. Smallest ready index that covers the shortfall (.smallest_ready_id_reserving_at_least)
SELECT "zoekt_indices"."id" FROM "zoekt_indices" WHERE "zoekt_indices"."zoekt_node_id" = <zoekt_node_id> AND "zoekt_indices"."state" = 10 AND "zoekt_indices"."reserved_storage_bytes" >= <bytes_to_free> ORDER BY "zoekt_indices"."reserved_storage_bytes" ASC, "zoekt_indices"."id" ASC LIMIT 1Plan: https://console.postgres.ai/gitlab/projects/gitlab-production-main/sessions/75426/commands/180136
(Index Scan on index_zoekt_indices_on_state_and_zoekt_node_id, top-N sort, 1 row, 180 ms cold, buffers: 9 hit / 1,580 read)
2. Largest ready reservations (.largest_ready_reservations(limit: 1000))
SELECT "zoekt_indices"."id", "zoekt_indices"."reserved_storage_bytes" FROM "zoekt_indices" WHERE "zoekt_indices"."zoekt_node_id" = <zoekt_node_id> AND "zoekt_indices"."state" = 10 AND "zoekt_indices"."reserved_storage_bytes" >= 1 ORDER BY "zoekt_indices"."reserved_storage_bytes" DESC, "zoekt_indices"."id" ASC LIMIT 1000Plan: https://console.postgres.ai/gitlab/projects/gitlab-production-main/sessions/75426/commands/180138
(Index Scan on index_zoekt_indices_on_state_and_zoekt_node_id, top-N sort, 1,000 rows, 67 ms cold, buffers: 1,525 hit / 1,012 read)
Why no index for the 180 ms cold case
- These queries run only for a node over the high watermark, at most hourly, and only with the flag on.
- An index on
(zoekt_node_id, state, reserved_storage_bytes)cuts query 1 to 17 buffers and query 2 to 3.1 ms (1,025 buffers), tested on a clone: https://console.postgres.ai/gitlab/projects/gitlab-production-main/sessions/75426/commands/180140 - It would add a migration and more maintenance on the frequent
reserved_storage_bytesbulk updates. That isn't worth it for this path, so it's left for a follow-up if needed.
How the plans were made
- Database Lab: snapshot from 2026-10-29 14:53 UTC.
- Query 1: a node with 11,195 ready indices, the shortfall of the 2026-10-29 eviction.
- Query 2: a node with 13,522 ready indices.
- Timing: both are first runs, so the timings are cold.
How to set up and validate locally
- Run
bin/rspec ee/spec/workers/search/zoekt/node_with_negative_unclaimed_storage_event_worker_spec.rb ee/spec/models/search/zoekt/index_spec.rb. - The
when zoekt_negative_storage_eviction_best_fit is disabledcontext keeps the existing examples for today's behavior.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.