Stop Zoekt node eviction from re-evicting on stale disk usage

What does this MR do and why?

Stops Zoekt node-level eviction on GitLab.com from repeatedly taking exact code search away from namespaces for nothing. This is part 1 of #635711.

On 2026-10-20, node gitlab-zoekt-26 crossed 75% disk because of temporary reindex data. Eviction then:

• removed 2,175 namespaces (50,099 projects), which lost exact code search for a median of about 4 hours and up to about 52 hours,
• placed every re-placed namespace (2,020) straight back on the same node, and evicted many of them again while it kept re-running on disk usage that hadn't dropped yet,
• achieved nothing: about 740 GiB of the spike was temporary, and the node ended up where it started.

Behind the zoekt_node_eviction_safeguards feature flag (gitlab_com_derisk, off by default), eviction now:

1. Evicts only from a node that has stayed over the high watermark (75%) for 8 hours, or right away when it's over the critical watermark (85%).
2. Puts the node in a cooldown after evicting. The node isn't evicted from again, and SelectionService places nothing new on it, until the indices from that eviction are gone (an index row is only destroyed once its repositories are deleted), or for at most 12 hours.
3. Picks ready indices smallest first by their actual used_storage_bytes on the node. Before, it used each namespace's whole repository_size. Indices that aren't ready yet are skipped: they only carry a placeholder used_storage_bytes, so evicting them frees nothing. On gprd every one of the ~34,000 pending indices has that placeholder.
4. Deletes only the replicas that have an index on that node. Before, it deleted every replica of the namespace.

The sustained and cooldown state is kept in Gitlab::Redis::SharedState with TTLs (Search::Zoekt::NodeEvictionState). With the flag off, eviction works exactly as before. Index-level eviction and negative-unclaimed-storage eviction are unchanged.

References

• #635711

Screenshots or screen recordings

Not applicable: background scheduling change with no UI.

How to set up and validate locally

1. Enable the flag in a Rails console:

   Feature.enable(:zoekt_node_eviction_safeguards)

2. Put a node in cooldown and check that placement skips it:

   node = Search::Zoekt::Node.for_search.online.first
   Search::Zoekt::NodeEvictionState.start_cooldown(node)
   Search::Zoekt::SelectionService.execute.nodes.map(&:id).include?(node.id) # => false
   Search::Zoekt::NodeEvictionState.end_cooldown(node)

3. Eviction only runs on GitLab.com (Gitlab::Saas.feature_available?(:exact_code_search)), so the eviction path itself is covered by ee/spec/services/search/zoekt/scheduling_service_spec.rb.

Database queries

Plans are from one gitlab-production-main Database Lab session (#72871) on a snapshot taken 2026-10-27 08:26 UTC. They run against node gitlab-zoekt-36 (ID 2000233), which has the most indices (15,209 of 120,004). Timings are cold-cache unless noted.

1. Eviction candidates (Search::Zoekt::Index.eviction_candidates)

Runs at most once per eviction, only after a node has stayed over the high watermark for 8 hours. The node is in cooldown afterwards.

SELECT "zoekt_indices"."namespace_id", "zoekt_indices"."used_storage_bytes", "zoekt_indices"."zoekt_replica_id"
FROM "zoekt_indices"
WHERE "zoekt_indices"."zoekt_node_id" = 2000233
  AND "zoekt_indices"."state" = 10
  AND "zoekt_indices"."zoekt_replica_id" IS NOT NULL
ORDER BY "zoekt_indices"."used_storage_bytes" ASC, "zoekt_indices"."id" ASC
LIMIT 5000

Plan: https://console.postgres.ai/gitlab/projects/gitlab-production-main/sessions/72871/commands/176882

Index Scan on index_zoekt_indices_on_state_and_zoekt_node_id, then an in-memory quicksort of the 3,643 matching rows (353 kB). 49 ms, 1,392 buffers (~10.9 MiB, 64 read).

2. Pending removal (node.indices.pending_removal.exists?)

Runs every 5 minutes for each node in cooldown.

SELECT 1 AS one FROM "zoekt_indices"
WHERE "zoekt_indices"."zoekt_node_id" = 2000233
  AND ("zoekt_indices"."zoekt_replica_id" IS NULL OR "zoekt_indices"."state" IN (225, 220, 230, 240))
LIMIT 1

Plan: https://console.postgres.ai/gitlab/projects/gitlab-production-main/sessions/72871/commands/176883

BitmapOr over index_zoekt_indices_on_replica_id and index_zoekt_indices_on_state_and_id. 4.6 ms, 12 buffers.

3. Nodes available for placement, excluding nodes in cooldown (SelectionService#fetch_available_nodes)

The existing Node.available_for_search_indexing query, plus id NOT IN (<cooldown node IDs>) when the flag is on and a node is in cooldown. The last_seen_at bound is one minute before the snapshot, so all 38 search nodes count as online on the clone.

SELECT zoekt_nodes.*, (zoekt_nodes.usable_storage_bytes - COALESCE(reserved_storage_bytes_total, 0)) AS unclaimed_storage_bytes
FROM "zoekt_nodes"
LEFT OUTER JOIN (
  SELECT "zoekt_nodes"."zoekt_node_id", sum(reserved_bytes) AS reserved_storage_bytes_total
  FROM ((SELECT 0, "zoekt_indices"."zoekt_node_id", sum(reserved_storage_bytes) AS reserved_bytes FROM "zoekt_indices" GROUP BY "zoekt_indices"."zoekt_node_id")) zoekt_nodes
  GROUP BY "zoekt_nodes"."zoekt_node_id"
) reserved_bytes ON zoekt_nodes.id = reserved_bytes.zoekt_node_id
WHERE (0 = ANY(services))
  AND "zoekt_nodes"."last_seen_at" >= '2026-10-27 08:25:11.000000'
  AND ((zoekt_nodes.usable_storage_bytes - COALESCE(reserved_storage_bytes_total, 0)) > 0)
  AND "zoekt_nodes"."id" NOT IN (2000233)
ORDER BY unclaimed_storage_bytes DESC

The new condition is only a filter (id <> 2000233) on the 38-row zoekt_nodes scan. Almost all of the cost is the existing reserved-bytes aggregate over zoekt_indices, which SelectionService already runs on master.

!264865 (merged) (merged) adds the covering index index_zoekt_indices_on_zoekt_node_id_include_reserved_bytes for that aggregate. Its post-deployment migration hasn't run on gprd yet, so the snapshot doesn't have it. The "with index" rows below create it on the clone with the migration's definition (create, 133 ms), then run only ANALYZE, so the clone keeps gprd's visibility map.

Run	Plan	Time	Buffers
Without index, cold	176880	4.5 s	100,769 (6,617 read)
Without index, warm	176881	64 ms	100,520
With index, production visibility map	176886	20 ms	4,105 (8,459 heap fetches)
With index, after VACUUM	176889	18 ms	505 (0 heap fetches)

Without the index, the aggregate reads all 120,004 zoekt_indices rows through index_zoekt_indices_on_zoekt_node_id_and_last_indexed_at and the heap. With it, the aggregate is a parallel index-only scan on the new index, with about 25 times fewer buffers. The "production visibility map" row is the closer estimate for gprd, since zoekt_indices is updated often.

4. Delete the evicted replicas

Runs once per eviction in slices of 100 replica IDs. Before, eviction deleted every replica of the evicted namespaces with Replica.for_namespace(...).each_batch.

DELETE FROM "zoekt_replicas" WHERE "zoekt_replicas"."id" IN (2907514, 2907942, ... 100 IDs)

Plan: https://console.postgres.ai/gitlab/projects/gitlab-production-main/sessions/72871/commands/176898

Index Scan on zoekt_replicas_pkey, 338 ms for 100 rows, 599 buffers (172 read). 189 ms of that is the ON DELETE SET NULL trigger of fk_zoekt_indices_on_zoekt_replica_id, the same work the old per-namespace delete did.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

🤖 Generated with Claude Code