# Plan This issue tracks two changes to Zoekt eviction, shipped as two MRs: 1. **Stop pointless evictions (MR 1):** https://gitlab.com/gitlab-org/gitlab/-/merge_requests/264628, behind `zoekt_node_eviction_safeguards` (rollout: https://gitlab.com/gitlab-org/gitlab/-/work_items/635761). Node-level eviction reacts to temporary disk usage, keeps re-running while space is being freed, and places evicted namespaces straight back on the same node. Details under part 1. 2. **Move instead of delete (MR 2):** an evicted namespace loses exact code search until it is placed and indexed again, because its replica is deleted first. Reworked after the team review in https://gitlab.com/gitlab-org/gitlab/-/work_items/635711#note_4075652774; needs design agreement before an MR. Details under part 2. # Part 1: Pointless evictions (MR 1) ## Summary On 2026-10-20, node `gitlab-zoekt-26` (ID `1000203`, ~3,023 GiB disk) crossed the 75% high watermark because of temporary data from a force reindex. Node-level eviction (`Search::Zoekt::SchedulingService#eviction`, GitLab.com only) then took exact code search away from 2,175 namespaces for hours, placed all of them back on the same node, and evicted many of them again. The node ended where it started: the eviction was not needed. ## What happened Node disk usage (`search_zoekt_node_storage_percent_used`, Grafana `mimir-gitlab-gprd`): | Time (UTC) | 10-20 17:00 | 10-20 23:00 | 10-21 04:00 | 10-21 06:00 | 10-21 12:00 | 10-23 | |---|---|---|---|---|---|---| | Disk used | ~1,600 GiB | 2,201 GiB | 2,273 GiB | 1,245 GiB | 1,481 GiB | ~1,540 GiB | <details><summary>Evidence from Kibana, Grafana, and a Database Lab clone of production</summary> - **Trigger:** 4,708 `force_index_repo` tasks ran on the node for 495 namespaces between 15:00 and 23:40 UTC on 2026-10-20. The disk grew by ~670 GiB. At 06:00 UTC on 2026-10-21 it dropped by ~1,030 GiB, but the evicted namespaces account for only ~290 GiB of that (the size of their new indices). So ~740 GiB was temporary data that cleared by itself. The node now holds the same namespaces at the same usage as before. - **Eviction runs:** 14 runs between 23:40 UTC on 2026-10-20 and 04:01 UTC on 2026-10-21 (`json.message: "Unassigning namespaces from node"`). Each selected ~3,500 namespaces, because the node kept reporting ~2.28 TiB used while files were still being deleted. - **Namespaces affected:** 2,175 distinct root namespaces (50,099 projects), from 67,163 `delete_repo` tasks. The "~3,500" per run included empty namespaces, which sort first and have nothing to delete. Later runs mostly re-selected the same namespaces; they did not evict a new set each time. - **Loop:** all 2,020 re-placed namespaces went back onto the same node, 21 to 109 minutes after 04:00 UTC (median 57). 17,060 deleted projects got new index tasks on the node during the eviction window itself, and ~14,900 projects had their repository on the node deleted twice or more (64,970 repository rows for 50,099 projects). - **Time without search:** re-indexing the evicted projects finished a median of ~4 hours after 04:00 UTC, p90 ~22 hours, p99 ~48 hours, worst ~52 hours. Group-level code search has no basic-search fallback (`lib/search/scopes.rb`), so without advanced search these namespaces had none. - **Size estimate:** eviction estimated ~470 GiB to free from `root_storage_statistics.repository_size`; the evicted namespaces' indices are ~290 GiB. - **gprd settings:** `zoekt_rollout_batch_size` 1, `zoekt_initial_indexing_limit` 1 (default), `zoekt_rollout_retry_interval` `1d`, `zoekt_default_number_of_replicas` 1 (default). All figures are aggregate counts; no namespace or project identifiers. </details> ## Causes 1. **It reacts to temporary disk usage.** It evicts as soon as the node reports 75%, including data that clears by itself. 2. **It re-runs while space is still being freed.** Disk usage only drops once the evicted repositories are deleted, hours later, so it keeps re-running every 5 minutes. Deleted replicas leave their index rows (`fk_zoekt_indices_on_zoekt_replica_id` is `ON DELETE SET NULL`), so it re-selects the same namespaces. 3. **It places evicted namespaces straight back on the same node.** Rollout prefers the node with the most unclaimed storage, which a node that just evicted usually is. 4. **It sizes the eviction from the wrong number:** the namespace's whole `repository_size`, not its index on this node. 5. **It deletes every replica of the namespace** (`Replica.for_namespace`), not only the one on the overfull node. ## Fix in MR 1 Behind `zoekt_node_eviction_safeguards` (`gitlab_com_derisk`, off by default; off keeps the current behavior): 1. Evict only after the node has stayed over the high watermark for 8 hours, or right away over the critical watermark (85%). 2. After evicting, put the node in a cooldown. It is not evicted from again, and rollout places nothing new on it, until the indices from that eviction are gone (an index row is destroyed only after its repositories), or for at most 12 hours. 3. Pick `ready` indices smallest first by their `used_storage_bytes` on the node. Indices that are not ready yet carry a placeholder `used_storage_bytes` (all ~34,000 `pending` indices on gprd do), so they are skipped. 4. Delete only the replicas with an index on that node. ## Not in MR 1 - Index-level eviction (`IndexToEvictEventWorker`) and negative-unclaimed-storage eviction (`NodeWithNegativeUnclaimedStorageEventWorker`, not limited to GitLab.com) are unchanged. - **What the temporary data was:** reindexing left ~740 GiB that only cleared hours later. This is likely indexer-side and needs its own look. - **Which namespaces to evict:** smallest first affects the most namespaces. "The largest namespaces that fit on another node" would affect far fewer. It needs eviction to know placement capacity, so it is a follow-up, and part 2 changes the trade-off anyway. - Repeated runs overwrite `EnabledNamespace#metadata['last_used_storage_bytes']` with 0 for already-evicted namespaces. Nothing reads it today. # Part 2: Move instead of delete (MR 2) ## Problem to solve When Zoekt evicts a namespace, it deletes the namespace's replica first and only then places and indexes a new one. With one replica per namespace (gprd uses the default of 1), the namespace has no exact code search the whole time: a median of ~4 hours and up to ~52 hours after the 2026-10-20 eviction. Today group-level code search falls back to advanced search where code is indexed in Elasticsearch. Once Elasticsearch is retired, nothing covers the gap. As a user of a namespace that Zoekt evicts to free space on a node, I want code search to keep working while my namespace moves to another node. ## Constraints found in the team review - **A move within the replica would go dark.** `ReplicaStateService` flips a `ready` replica to `pending` as soon as any of its indices is not ready, and `Replica.search_enabled?` requires a `ready` replica. Adding a new index to the same replica takes the namespace out of search for the whole move. - **Node eviction deletes every replica of a namespace** (`Replica.for_namespace`). MR 1 fixes this behind its flag. - **There are three eviction triggers:** node-level (GitLab.com only), index-level (`IndexToEvictEventWorker`), and negative unclaimed storage (`NodeWithNegativeUnclaimedStorageEventWorker`, all instances). All three delete first. ## Proposal Move with an extra replica: build a new replica on another node while the old one keeps serving, and delete the old one only once the new one is `ready`. 1. **Add a replica.** When a namespace is chosen for eviction, give it a temporary extra replica on a node other than the overfull one. The old replica stays `ready` and keeps serving search. 2. **Index it ahead of the queue.** Prioritize the new replica's indices in `SchedulingService#initial_indexing` ahead of never-indexed `pending` indices (about 34,000 on gprd, mostly trials, started at 1 per node per minute). 3. **Swap.** Once the new replica is `ready`, delete only the old replica, the one with the index on the overfull node. 4. **Keep a hard stop.** Delete-first stays only as a last resort, when a node reaches the critical watermark or no other node has room. ## Open questions - **Headroom:** unclaimed storage across gprd was ~0.77 TiB on 2026-10-23, with at most ~52 GiB on one node. Moving needs room for the second copy; without it, eviction falls back to delete-first. - **New projects mid-move:** they must be indexed into both replicas, or into the new one before the swap. - **Replica count:** `EnabledNamespace.each_batch_with_mismatched_replicas` handles namespaces with too many replicas. It must not delete either replica while a move is in progress. - **Single-replica delete path:** an explicit way to delete only the old replica, not every replica of the namespace. - **All three triggers:** decide which use moves, including self-managed instances for negative unclaimed storage. - **Trigger point:** moves take hours, so they may need to start before the node reaches the hard stop. ## Success criteria - A namespace moved off a full node keeps returning exact code search results the whole time. - A metric shows how long each move takes and how often eviction falls back to delete-first. ## Related - https://gitlab.com/gitlab-org/gitlab/-/work_items/634144: Zoekt capacity planning - https://gitlab.com/gitlab-org/gitlab/-/work_items/635761: rollout of `zoekt_node_eviction_safeguards`