## Summary `Search::Zoekt::PlanningService` places namespaces on nodes using disk alone. The resource that actually saturates a Zoekt node in production is **webserver resident memory**, and resident memory is not proportional to index size on disk. A node can sit at roughly a quarter of its disk capacity — nowhere near any watermark — while its webserver carries well over a hundred GiB resident. The planner cannot see this, and the watermark machinery that would normally trigger rebalancing is a pure disk ratio (`ee/app/models/search/zoekt/node.rb:285`), so it can never fire on a memory-only imbalance. The telemetry needed to fix this is **already collected**. Per-node webserver `rss_bytes` and `shards_loaded` arrive on every heartbeat (`ee/lib/api/internal/search/zoekt.rb:24`, `ee/lib/api/internal/search/zoekt.rb:90`) and are persisted into `zoekt_nodes.metadata` (`ee/app/models/search/zoekt/node.rb:336`). They are consumed by exactly one thing: human-readable CLI output in `gitlab:zoekt:info` (`ee/app/services/search/zoekt/info_service.rb:286`). **The gap is consumption, not collection.** This issue proposes adding memory as a first-class planning factor: a per-index memory-cost estimate, a per-node memory budget alongside the storage budget, and a maximum-memory-per-index constraint that forces a hot namespace to split across nodes when disk alone would keep it on one. Originating from a self-managed customer escalation: [gitlab-com/request-for-help#5304](https://gitlab.com/gitlab-com/request-for-help/-/issues/5304). ## Problem ### Disk and resident memory are not proportional The published sizing guidance in [the Zoekt memory architecture docs](https://docs.gitlab.com/integration/zoekt/#memory-architecture) gives a rule of thumb of roughly 256 GB of index disk to 32 GiB of webserver memory — about a 12% resident-to-disk ratio. A heavily-searched namespace on a self-managed cluster was observed at roughly **49%**, four times that. The ratio is driven by search traffic and by which shards the webserver has touched, not by index size, so two indices of identical size on disk can differ by an order of magnitude in resident cost. Any planner that reasons only about bytes on disk is structurally incapable of predicting or reacting to this. ### The observed shape of the failure On the reporting cluster (two nodes, round figures): | | Node 0 | Node 1 | | --- | --- | --- | | Disk used | ~27% | ~6% | | Webserver resident | ~136 GiB | ~5 GiB | One dominant top-level namespace occupied a single index. Because that index fit comfortably within one node's unclaimed disk, the planner assigned it to one node and never had reason to do otherwise — a ~27x memory imbalance between two nodes whose disk usage differs by a factor of four, neither remotely close to a watermark. This is not a disk imbalance that happens to have a memory symptom; it is a memory imbalance the disk-only planner cannot represent. Downstream, the resulting per-pod memory footprint exceeded the operator's own scheduling limits. ### The workaround that exists today, and why it is unsatisfying A remedy exists today and works — a customer has validated it on staging: 1. Shrink the per-node index PVCs to below the hot namespace's `reserved_storage_bytes`. 2. Reindex the namespace from scratch. With smaller nodes, `find_best_node` (`ee/app/services/search/zoekt/planning_service.rb:186`) can no longer find a single node whose `unclaimed_storage_bytes` covers the whole namespace, so the packing loop in `assign_project_to_index` (`ee/app/services/search/zoekt/planning_service.rb:189`) rolls over into a new index on a different node; fanning out the shards fans out the resident memory. Three things make this unacceptable as the long-term answer: - **It requires a full reindex.** The project-to-index mapping is fixed at plan time via `project_namespace_id` ranges (`ee/app/services/search/zoekt/planning_service.rb:228`); there is no re-partition path that preserves existing shards. - **It requires a maintenance window.** PVCs cannot be shrunk in place on most storage classes, so this is a destroy-and-recreate on the index volumes. - **It tunes a disk knob to achieve a memory outcome.** The operator lies to the planner about how much disk they have to provoke a split the planner has no vocabulary to request. It is fragile: grow the namespace, add a node, or change the buffer factor, and the coincidence that produced the split can evaporate. ## Current behavior ### The planner considers only disk `ee/app/services/search/zoekt/planning_service.rb` contains no memory term. Only three quantities influence placement: `node[:unclaimed_storage_bytes]`, the sole capacity figure carried into the planner's node model (`ee/app/services/search/zoekt/planning_service.rb:45`); `scaled_size(stats)` = `stats.repository_size * buffer_factor` (`ee/app/services/search/zoekt/planning_service.rb:249`), the buffer factor being a disk-to-disk ratio from `used_storage_bytes / repository_size` across ready indices (`ee/app/models/search/zoekt/index.rb:165`); and `max_storage_bytes` per simulated index, seeded from the node's unclaimed disk (`ee/app/services/search/zoekt/planning_service.rb:219`). `find_best_node` returns the first non-exhausted node with enough unclaimed disk (`ee/app/services/search/zoekt/planning_service.rb:186`), and projects pack into the trailing index while `required_storage_bytes + project_size <= max_storage_bytes` (`ee/app/services/search/zoekt/planning_service.rb:192`). A new index — and therefore a possible move to a new node — is created only when that disk inequality fails, so no expression exists under which a namespace splits for any reason other than running out of disk. ### Watermarks are a pure disk ratio `storage_percent_used` is `used_bytes / total_bytes` (`ee/app/models/search/zoekt/node.rb:285`) and drives all three watermark predicates (`ee/app/models/search/zoekt/node.rb:277`, `ee/app/models/search/zoekt/node.rb:281`) against limits of 0.6 / 0.75 / 0.85 (`ee/app/models/search/zoekt/node.rb:13`); node capacity is `free_bytes + indexed_bytes` (`ee/app/models/search/zoekt/node.rb:351`) and unclaimed capacity `usable_storage_bytes - reserved_storage_bytes` (`ee/app/models/search/zoekt/node.rb:291`). At ~27% disk none can fire, no matter how much resident memory the node carries. ### Memory telemetry already reaches Rails and is display-only This is the important distinction, and it is easy to state wrongly. GitLab **does** have per-node memory data for Zoekt. The heartbeat's `:process_metrics` shared params block declares `rss_bytes` (`ee/lib/api/internal/search/zoekt.rb:24`) alongside mmap counters, restart counters, and uptime; the `heartbeat` endpoint accepts a `process_health` hash with `indexer` and `webserver` sub-hashes, the `webserver` one additionally carrying `shards_loaded` (`ee/lib/api/internal/search/zoekt.rb:90`). `assign_metadata` stores the whole payload at `metadata['process_health']` and stamps `webserver_last_seen_at` (`ee/app/models/search/zoekt/node.rb:331`, `ee/app/models/search/zoekt/node.rb:336`), and the persisted schema includes indexer RSS (`ee/app/validators/json_schemas/zoekt_node_metadata.json:41`), webserver RSS (`ee/app/validators/json_schemas/zoekt_node_metadata.json:68`), and `shards_loaded` (`ee/app/validators/json_schemas/zoekt_node_metadata.json:74`). Outside specs, the only consumers of `rss_bytes` are the CLI formatters `ee/app/services/search/zoekt/info_service.rb:269` (indexer) and `ee/app/services/search/zoekt/info_service.rb:286` (webserver), which render it as a human-readable string; `shards_loaded` likewise reaches only `ee/app/services/search/zoekt/info_service.rb:289`. No scheduler, planner, eviction path, or routing decision reads it, and it is absent from the structured node payload used for logging and metadata (`ee/app/models/search/zoekt/node.rb:258`), which carries disk and concurrency fields and no memory field. ### There is one memory-adjacent guardrail, and it is not RSS `Search::Zoekt::ProcessHealth` excludes a node from **search routing** when restarts exceed a setting, when `mmap_current / mmap_max >= 0.95`, or when the webserver heartbeat is stale (`ee/lib/search/zoekt/process_health.rb:20`, `ee/lib/search/zoekt/process_health.rb:42`), wired in via `Node.searchable` (`ee/app/models/search/zoekt/node.rb:144`). It is a reactive exclusion applied at query time to a node already in trouble; it does not read `rss_bytes`, has no influence on placement, and cannot prevent a node from being handed a namespace that will make it exhaust memory. It is also gated on all online nodes running at least `MIN_VERSION = '1.16.0'` (`ee/lib/search/zoekt/process_health.rb:6`, `ee/lib/search/zoekt/process_health.rb:13`), so on clusters with older nodes these signals are inert. ### No memory column exists on either table `zoekt_nodes` has `used_bytes`, `total_bytes`, `indexed_bytes`, and `usable_storage_bytes`; `zoekt_indices` has `reserved_storage_bytes`, `used_storage_bytes`, and `watermark_level`. Every byte-denominated column on both is disk, and no per-index estimate of resident memory cost exists in the schema. ### Rebalancing is SaaS-gated and would shed the wrong thing `eviction` returns early unless `::Gitlab::Saas.feature_available?(:exact_code_search)` (`ee/app/services/search/zoekt/scheduling_service.rb:181`), so on self-managed — where this was reported — the proactive rebalancer does not run. Even where it does, it selects nodes by `watermark_exceeded_high?` (`ee/app/services/search/zoekt/scheduling_service.rb:185`) and evicts namespaces **smallest-first** by repository size (`ee/app/services/search/zoekt/scheduling_service.rb:206`), so where memory pressure comes from one large hot namespace among thousands of tiny ones, that ordering sheds the tiny ones and leaves the cause in place. ### Replicas duplicate, they do not shard Raising `zoekt_default_number_of_replicas` (`ee/app/models/search/zoekt/settings.rb:141`) is not a mitigation. Each replica is a full copy of the namespace's project set, and `add_replica_plan` merges each replica's node ids into `@exhausted_node_ids` (`ee/app/services/search/zoekt/planning_service.rb:238`) precisely so replicas land on different nodes; more replicas multiplies total resident memory across the fleet rather than dividing it. ### A replica can hold at most one index per non-exhausted node This constraint, separate from the replica-level exhaustion above, governs how finely a namespace can be split. Inside a replica, every time the planner creates a new index for a project after the replica's first, it permanently marks the chosen node exhausted: `@exhausted_node_ids.add(node[:id]) if last_index` (`ee/app/services/search/zoekt/planning_service.rb:205`). `find_best_node` skips exhausted nodes (`ee/app/services/search/zoekt/planning_service.rb:186`), so once every node has taken one index for this replica it returns `nil` and `assign_to_node` records `:node_unavailable` (`ee/app/services/search/zoekt/planning_service.rb:181`) for every remaining project. The derived ceiling: **a split cannot produce more indices per replica than there are non-exhausted nodes.** Node count, not `MAX_INDICES_PER_REPLICA`, is the binding limit in practice. With two nodes, then, the best achievable balance for a hot namespace is 50/50. A memory-aware planner makes the imbalance representable, plannable, and diagnosable — but a cluster may still need more nodes to bring any single pod under a given per-pod memory threshold. This change is necessary for that outcome; on a two-node cluster it is not sufficient by itself. ### The per-replica index cap (secondary) `MAX_INDICES_PER_REPLICA = 10` (`ee/app/models/search/zoekt.rb:8`) is threaded through `RolloutService` (`ee/app/services/search/zoekt/rollout_service.rb:11`) into the planner. When a replica reaches the cap the planner records an `:index_limit_exceeded` error and **breaks out of the project loop** (`ee/app/services/search/zoekt/planning_service.rb:144`), so every remaining project in that namespace is silently left unplanned. On clusters with ten or more nodes this becomes the binding ceiling instead of node exhaustion; below that, node count binds first. ## Proposal Give the planner a memory dimension that parallels the existing storage dimension, then add one new constraint — a maximum memory per index — that can force a split the disk math would not produce. ### Step 1 (recommended first increment): estimated memory cost + per-node memory budget + max-memory-per-index **1a. A per-index memory-cost estimate, computed from what the planner has.** Define, at plan time: ``` estimated_memory_bytes = repository_size * memory_ratio ``` where `repository_size` is the same `project.statistics` figure the planner already reads (`ee/app/services/search/zoekt/planning_service.rb:141`) and `memory_ratio` is an instance-level setting with a conservative default, calibrated **`repository_size` → resident bytes end to end**. The base matters, and the obvious alternative does not work. `used_storage_bytes` is populated only by `refresh_used_storage_bytes` from `zoekt_repositories.sum(:size_bytes)` (`ee/app/models/search/zoekt/index.rb:252`), falling back to `DEFAULT_USED_STORAGE_BYTES = 1.kilobyte` (`ee/app/models/search/zoekt/index.rb:10`) when that sum is zero; on the `:create` path the planner invents indices with no repository rows yet, so it is absent or 1 KiB precisely when the forced-split decision must be made. `repository_size` is available for every project on every path. `memory_ratio` is deliberately **not** multiplied by `buffer_factor`: it maps raw repository bytes directly to resident bytes, so it is a different number from any disk-to-disk ratio in the codebase and must be calibrated as such. In shape it mirrors the `buffer_factor` pattern — a global ratio, defaulted, cached, applied uniformly (`ee/app/models/search/zoekt/index.rb:11`, `ee/app/models/search/zoekt/index.rb:126`). Step 2's estimated-vs-observed comparison operates on **already-placed** indices and may use `used_storage_bytes * observed_memory_ratio` there, since those indices do have a populated column; that second ratio is a different quantity from the plan-time `memory_ratio` and the two must not be conflated in the implementation or the docs. Deliberately **not** modelled at first: per-namespace ratios, query-rate weighting, shard-count weighting — those need calibration data we do not yet aggregate. **1b. A per-node memory budget.** Add `unclaimed_memory_bytes` to the planner's node model beside `unclaimed_storage_bytes` (`ee/app/services/search/zoekt/planning_service.rb:45`), computed as `node_memory_budget_bytes - sum(estimated_memory_bytes of assigned indices)`, where `node_memory_budget_bytes` comes from a **separate** instance setting (the per-node memory budget — see the options table below). Extend `find_best_node` (`ee/app/services/search/zoekt/planning_service.rb:186`) to require *both* budgets to cover the candidate project, and decrement both in `assign_project_to_index` (`ee/app/services/search/zoekt/planning_service.rb:209`). **1c. A maximum memory per index, with the rollover node chosen correctly.** Add `max_memory_bytes` to the simulated index alongside `max_storage_bytes` (`ee/app/services/search/zoekt/planning_service.rb:219`) and extend the packing condition (`ee/app/services/search/zoekt/planning_service.rb:192`) so a project only joins the trailing index when it fits under *both* ceilings. `max_memory_bytes` is a **constant** for the deployment: a fixed fraction of 1b's per-node memory budget, i.e. `node_memory_budget_bytes / concurrency_factor`, where `concurrency_factor` is the number of namespaces a node should carry concurrently — a **hardcoded constant with a stated default**, not a third instance setting, so one fewer knob to misconfigure, promotable later. It is explicitly **not** seeded from the node's *current remaining* memory the way `max_storage_bytes` is seeded from remaining unclaimed disk at `:219`: a ceiling that varies with whatever a node happens to have left is not explainable to an operator, and the goal — a single namespace cannot claim a whole node's memory — is a property of a fixed fraction. **The node-selection ordering is the load-bearing part of this mechanism, and today's control flow gets it wrong for memory.** `assign_to_node` calls `find_best_node` **first** (`ee/app/services/search/zoekt/planning_service.rb:175`), before any packing decision; `assign_project_to_index` then evaluates the packing condition at `:192`; and only if it fails is a new index created — on the node `find_best_node` already picked — with `@exhausted_node_ids.add(node[:id]) if last_index` firing at `:205`, **after** the index has been placed there. In the disk-driven case this is harmless only by coincidence: `max_storage_bytes` is seeded from exactly the node's unclaimed disk at `:219` and `:209` decrements the same node's `unclaimed_storage_bytes` in lockstep, so by the time the packing condition fails the node has ~0 unclaimed disk and `find_best_node` skips it anyway. A memory ceiling deliberately breaks that coupling — that is the entire point of it. When `max_memory_bytes` binds the node still has abundant unclaimed disk, and 1c's own sizing guarantees it passes the 1b check too: with any `concurrency_factor` above 1, one index reaching `max_memory_bytes` leaves the node's overall memory budget nowhere near exhausted. `find_best_node` returns the same node and the naive implementation puts a second index on that **same node** — zero memory relief, while looking like it worked. The requirement: > When the memory ceiling is the reason for a rollover, the current node must be > excluded from candidate selection **before** the new index is placed. **Both** changes are needed. The node must be added to `@exhausted_node_ids` at the moment the memory ceiling is identified as the reason for the roll — not conditionally at `:205` after the fact — so the exclusion persists for the rest of the replica and the N-node/N-way ceiling continues to hold. And candidate selection must happen *after* the packing test rather than before it (`:175` runs before `:192` today), so the current roll honours that exclusion. Adding to the set without re-selecting leaves `simulate_index` at `:199` on the same node; re-selecting without adding to the set lets a later index return to it. With both: on two nodes, index 1 lands on node A, index 2 on node B, then `find_best_node` returns `nil` and **1d** degrades — remaining projects keep packing into index 2 past the ceiling with a warning — exactly two indices on distinct nodes, satisfying criteria 1, 4 and 5. The ceiling is enforced on the packing decision at `:192`, guarded by `last_index &&`, so a replica's *first* index and any single project whose estimated memory alone exceeds the ceiling are not bounded by it; a project is indivisible because indices are `project_namespace_id` ranges (`ee/app/services/search/zoekt/planning_service.rb:228`). `max_memory_bytes` is therefore a best-effort bound on packing, not an invariant on index memory cost. **1d. Memory-driven placement failure must degrade, not void the plan.** A hard requirement of the proposal, not a nicety. `PlanningService#plan` partitions on errors: `all_plans.partition { |plan| plan[:errors].present? }` (`ee/app/services/search/zoekt/planning_service.rb:22`). **Any** accumulated error moves the namespace's *entire* plan into `failures`, and `ProvisioningService` handles `plan[:failures]` by calling `update_enabled_namespace` (`ee/app/services/search/zoekt/provisioning_service.rb:19`, `ee/app/services/search/zoekt/provisioning_service.rb:159`), which sets `last_rollout_failed_at` and creates no indices for that namespace at all. So a memory ceiling that cannot be satisfied on the available nodes must **not** accumulate an error: it must fall back to the disk-only placement decision for that project and log a warning. Fragmenting suboptimally is acceptable; un-provisioning the hot namespace entirely is strictly worse than today's behaviour, and is what an unguarded implementation produces. **Why this first.** It is the smallest change that makes the reported failure impossible to reproduce, it reuses the buffer-factor and unclaimed-bytes patterns already in the planner rather than inventing a new capacity model, and it behaves identically on self-managed and SaaS because it lives entirely in the planning path — not behind the `Gitlab::Saas.feature_available?` gate that excludes self-managed from `eviction` (`ee/app/services/search/zoekt/scheduling_service.rb:181`). ### Where the memory budget comes from Three options; not mutually exclusive, and they should land in this order. | Option | Mechanism | Pros | Cons | | --- | --- | --- | --- | | **A. Configured** | New instance setting for per-node memory budget, in the existing `SETTINGS` hash (`ee/app/models/search/zoekt/settings.rb:34`) | Works on every deployment; operator states their real constraint; no new node-side work | Manual; a stale value silently misplans; assumes homogeneous nodes | | **B. Reported by the node** | Extend the heartbeat's `:process_metrics` block (`ee/lib/api/internal/search/zoekt.rb:18`) with a memory limit read from the cgroup, persisted like `rss_bytes` is today | Accurate, per-node, self-maintaining; heterogeneous nodes handled correctly | Requires an indexer-side change and a version floor; unreported on older nodes | | **C. Observed** | Derive from reported webserver `rss_bytes` history | No new telemetry at all — the data is already in `zoekt_nodes.metadata` | Measures *usage*, not *capacity*; a node under-loaded today looks like it has headroom it does not have | **Recommendation: A first, then B, with C used only for calibration.** A configured budget is the only option that works on day one for every deployment including self-managed, and it makes the constraint auditable. B is the correct end state, to follow as a version-gated refinement defaulting back to A when absent — the same shape as the `ProcessHealth::MIN_VERSION` gate (`ee/lib/search/zoekt/process_health.rb:6`). C should not set the budget; it is the right source for calibrating the *ratio* in 1a. ### Step 2: calibrate the ratio from observed RSS Once 1a ships with a fixed default, close the loop. For nodes reporting webserver `rss_bytes` and `shards_loaded`, compare observed resident memory against the sum of `estimated_memory_bytes` for indices resident on that node and surface the discrepancy — first as a logged metric and in `gitlab:zoekt:info` next to the existing RSS line (`ee/app/services/search/zoekt/info_service.rb:293`), later as an automatic adjustment to the global ratio computed the way `compute_global_buffer_factor` already computes the disk ratio (`ee/app/models/search/zoekt/index.rb:165`). Calibration stays separate because a self-tuning ratio that is wrong early is worse than a fixed conservative one; we should see the observed distribution before automating against it. ### Node exhaustion and the index cap Both ceilings on a memory-forced split are derived under Current behavior: node exhaustion at `ee/app/services/search/zoekt/planning_service.rb:205`, and `MAX_INDICES_PER_REPLICA = 10` (`ee/app/models/search/zoekt.rb:8`) as a secondary ceiling binding only at ten or more nodes. This issue does **not** propose relaxing `:205` — a substantial design change touching search fan-out and per-index overhead, listed under Out of scope. Three requirements follow, part of this proposal and not deferred: 1. **Memory-driven node exhaustion must be attributable.** When a namespace runs out of non-exhausted nodes and memory drove the extra indices, the accumulated error or warning (`ee/app/services/search/zoekt/planning_service.rb:258`) must say so distinctly from disk-driven exhaustion: the operator needs to know whether to add disk, memory, or a node — usually the last, on a small cluster. 2. **This must be visible in the plan log before it is felt in provisioning.** `RolloutService` logs the full plan payload unconditionally at `ee/app/services/search/zoekt/rollout_service.rb:44`, before the `dry_run` early return at `ee/app/services/search/zoekt/rollout_service.rb:45`, so it is emitted on every real run; the memory fields — per-index estimated cost, per-node budget before and after, any 1d degradation warning — must appear in it. 3. **Whether the cap should rise is an open design question, not an assumption.** `max_indices_per_replica` is already injectable per call (`ee/app/services/search/zoekt/rollout_service.rb:11`), so raising it is mechanically easy — but 10 exists for reasons (fan-out cost per search, per-index overhead) that need restating first. This issue does not propose raising it, and below ten nodes it would change nothing. ### Self-managed The reporting deployment is self-managed, and Step 1 lives entirely in the planning path, which runs there with no SaaS gate anywhere in the chain: the `RolloutWorker` cron runs every ten minutes (`ee/config/schedule.yml:345`) and calls `RolloutService.execute(dry_run: false, ...)` (`ee/app/workers/search/zoekt/rollout_worker.rb:30`), which resolves nodes via `SelectionService` (`ee/app/services/search/zoekt/selection_service.rb:37`) and then calls `PlanningService.plan` (`ee/app/services/search/zoekt/rollout_service.rb:39`). Two adjacent mechanisms are easy to mistake for this path. `auto_index_self_managed` (`ee/app/services/search/zoekt/scheduling_service.rb:268`) only inserts `EnabledNamespace` rows for root namespaces that lack them — it enrolls namespaces, it does not invoke the planner. `saas_rollout` (`ee/app/services/search/zoekt/scheduling_service.rb:111`) is the SaaS-gated sibling (`ee/app/services/search/zoekt/scheduling_service.rb:113`), not this path. No part of Step 1 may sit behind `Gitlab::Saas.feature_available?`, and any follow-up hooking into `eviction` inherits that gate (`ee/app/services/search/zoekt/scheduling_service.rb:181`) and is SaaS-only until the gate is addressed separately. ## Scope - A per-index estimated memory cost derived from `repository_size` and a configurable `memory_ratio`. - A per-node memory budget, configured via its own instance setting, carried into the planner's node model alongside `unclaimed_storage_bytes`. - A maximum-memory-per-index constraint in the planner that forces a namespace to split across nodes when its estimated resident cost exceeds the ceiling, including the node-selection reordering required so the new index lands on a **different** node than the one the ceiling was hit on. - Graceful degradation: a memory-driven inability to place a project falls back to the disk-only decision for that project and logs a warning, and must never accumulate an error that voids the namespace plan at `ee/app/services/search/zoekt/planning_service.rb:22`. - Distinct, attributable planner errors and warnings when a memory-driven split runs out of non-exhausted nodes (`ee/app/services/search/zoekt/planning_service.rb:205`), or reaches `MAX_INDICES_PER_REPLICA`. - Memory fields in the unconditional rollout plan log payload (`ee/app/services/search/zoekt/rollout_service.rb:44`). - `create_empty_replica` (`ee/app/services/search/zoekt/planning_service.rb:157`) carries the new node fields through unchanged — an empty replica has no estimated memory cost, so no memory ceiling applies to it. The destroy branch (`ee/app/services/search/zoekt/planning_service.rb:108`) never calls `simulate_index` or `find_best_node` and is untouched by this change. - Surfacing estimated vs. observed memory in `gitlab:zoekt:info`, using the `rss_bytes` and `shards_loaded` values already persisted. - Documentation of both new settings, of the `concurrency_factor` constant and its default, and of how the estimate is derived. - Behaviour must be identical on self-managed and SaaS. ## Out of scope - Changing `MAX_INDICES_PER_REPLICA` from 10. - Relaxing the intra-replica node exhaustion at `ee/app/services/search/zoekt/planning_service.rb:205` to allow more than one index per node per replica. This would raise the achievable split granularity on small clusters, but it is a substantial design change with its own fan-out and overhead consequences and needs a separate issue. - A memory-denominated companion to the disk watermark predicates (`ee/app/models/search/zoekt/node.rb:277`) and its surfacing in `metadata_json` (`ee/app/models/search/zoekt/node.rb:258`). This is a worthwhile follow-up — it would make memory pressure detectable by the same machinery that detects a full disk — but Step 1 is already large and this touches node-level predicates rather than the planner. - An operator-facing rake task for a planning dry run. The unconditional plan log (`ee/app/services/search/zoekt/rollout_service.rb:44`) is the visibility this issue relies on; `dry_run: true` is only the `DEFAULT_OPTIONS` value (`ee/app/services/search/zoekt/rollout_service.rb:12`) and the sole caller passes `dry_run: false` (`ee/app/workers/search/zoekt/rollout_worker.rb:30`), so no operator entry point for a dry run exists today. - Memory-driven eviction or rebalancing of already-placed indices. Placement first; the eviction path has a SaaS gate and a smallest-first ordering that both need separate work. - Repartitioning an existing namespace without a reindex. - Node-side changes to the Zoekt indexer or webserver. Step 1 uses only telemetry that exists today; the cgroup-reported memory limit (Option B) is a follow-up. - Per-namespace or query-rate-weighted memory ratios. - Automatic self-tuning of the ratio. Step 2 delivers the measurement; acting on it automatically is a follow-up. - Changing the disk watermark thresholds (`ee/app/models/search/zoekt/node.rb:13`) or the buffer factor (`ee/app/models/search/zoekt/index.rb:11`). ## Acceptance criteria 1. A planner unit test constructs two nodes with ample unclaimed disk and one namespace **with at least two projects** whose estimated memory cost exceeds `max_memory_bytes`, and asserts that **every** index in the resulting plan for that namespace has a distinct `node_id` — not merely that more than one node appears. A plan placing two indices on node A and one on node B must fail this test. The same fixture with the memory constraint disabled produces a single-node plan. 2. A planner unit test asserts that when both budgets are satisfied, placement is byte-for-byte identical to today's disk-only plan — no behaviour change on clusters that are not memory-constrained. 3. `find_best_node` (`ee/app/services/search/zoekt/planning_service.rb:186`) rejects a node whose remaining memory budget is insufficient even when its `unclaimed_storage_bytes` is sufficient, proven by a test asserting on the error from `assign_to_node` (`ee/app/services/search/zoekt/planning_service.rb:181`) — not the one from `create_empty_replica` (`ee/app/services/search/zoekt/planning_service.rb:165`) — carrying a memory-specific `type` distinct from `:node_unavailable` rather than overloading `details`. 4. When a memory-forced split exhausts every available node (`ee/app/services/search/zoekt/planning_service.rb:205`), the emitted diagnostic is distinguishable from disk-driven node exhaustion by its `type` or `details`, asserted by test. A test on a two-node fixture asserts the namespace is split into exactly two indices, one per node, and that the planner reports it could not split further because nodes were exhausted. 5. A namespace whose memory ceiling cannot be satisfied on the available nodes is still **fully planned** — every project placed by falling back to the disk-only decision, with a warning logged — and does **not** appear in `plan[:failures]` (`ee/app/services/search/zoekt/planning_service.rb:22`), asserted by test. `last_rollout_failed_at` must not be set for that namespace (`ee/app/services/search/zoekt/provisioning_service.rb:159`). 6. The unconditional plan log (`ee/app/services/search/zoekt/rollout_service.rb:44`) — which fires before the `dry_run` early return at `ee/app/services/search/zoekt/rollout_service.rb:45`, and therefore on normal runs — includes, per namespace, the estimated memory cost per index, the per-node memory budget before and after, and any degradation warning from 1d, asserted by test on the structured payload. 7. With the per-node memory budget setting unset, planning produces the same plan as before this change, asserted by test: the feature is inert until the budget is set. 8. A node whose `metadata['process_health']` is absent, or present but with no `webserver` key, still yields a memory-aware plan with no nil-arithmetic error and without being excluded from candidate selection, asserted by test — guarding the Option B path, where reported memory becomes an input. 9. `gitlab:zoekt:info` prints, per node, the sum of estimated memory cost for its indices next to the existing observed webserver RSS line (`ee/app/services/search/zoekt/info_service.rb:293`). 10. No `Gitlab::Saas.feature_available?` call exists anywhere in the planning call chain — `RolloutWorker` (`ee/app/workers/search/zoekt/rollout_worker.rb:30`) → `RolloutService` (`ee/app/services/search/zoekt/rollout_service.rb:39`) → `PlanningService` — enforced by a test that fails if one is introduced into the new code paths. A regression guard, not a stub: the chain is ungated today and must stay so. 11. Documentation at the Zoekt memory architecture page states both new settings and their defaults, the `concurrency_factor` constant and its default, how the estimate is computed, that the plan-time `memory_ratio` and the Step 2 observed ratio are different numbers, and that the published disk-to-memory rule of thumb is a default rather than a guarantee. 12. Documentation states plainly that on a cluster with N nodes a namespace can be split at most N ways, so relieving a per-pod memory threshold may require adding nodes and not only configuring the ceiling. ## Risks and constraints **Reindex cost.** A memory-aware plan that splits a namespace differently from its current layout implies moving projects between indices. The project-to-index mapping is a `project_namespace_id` range fixed at plan time (`ee/app/services/search/zoekt/planning_service.rb:228`) and there is no in-place repartition, so re-planning an existing hot namespace means a reindex and a maintenance window — the same cost as today's workaround. This change makes future placements correct; it does not make the current migration free, and the docs should say so. **Estimate error, over-estimating.** If `memory_ratio` is set too high, or `max_memory_bytes` too low, the planner fragments namespaces that did not need it: more indices per replica, higher search fan-out, more per-index overhead. The severe cost is different — once every node is exhausted for a replica (`ee/app/services/search/zoekt/planning_service.rb:205`), the accumulated errors move the namespace's *entire* plan into `failures` (`ee/app/services/search/zoekt/planning_service.rb:22`) and it stops being provisioned at all. 1d states that mechanism in full; it is why 1d is mandatory and criterion 5 asserts on it directly. A conservative default does not on its own protect against this: the trigger is node count (see the node-exhaustion ceiling under Current behavior), not the ratio being wildly wrong. Two different settings limit the exposure — a conservative default `memory_ratio` bounds the overshoot, and the per-node memory budget has no default at all, so the constraint stays disengaged until an operator configures it (criterion 7). Neither substitutes for 1d. **Estimate error, under-estimating.** If the ratio is too low, the planner believes a node has memory headroom it does not have and we reproduce the reported failure while appearing to have fixed it — the worse direction, because it is silent. Step 2's estimated-vs-observed comparison is the detection mechanism and should not lag far behind Step 1. **Ratio is a single global number.** The real ratio varies per namespace with search traffic, so a global ratio will be wrong for both the hottest and coldest namespaces at once — a deliberate first-increment simplification, not a claim one number suffices. **Node count is the real ceiling.** A namespace cannot be split more ways than there are non-exhausted nodes (`ee/app/services/search/zoekt/planning_service.rb:205`) — on two nodes 50/50 is the best case however `max_memory_bytes` is configured (see Current behavior). Criteria 4, 5 and 6 make the ceiling loud, not absent. **Clusters with no memory telemetry.** `ProcessHealth` requires all online nodes at `MIN_VERSION = '1.16.0'` (`ee/lib/search/zoekt/process_health.rb:6`, `ee/lib/search/zoekt/process_health.rb:13`), and `process_health` is optional on the heartbeat (`ee/lib/api/internal/search/zoekt.rb:84`). Where no node reports it, `metadata['process_health']` is absent and there is nothing to calibrate against, so Step 1 must not depend on reported RSS — the configured budget and the derived estimate suffice on their own, and Step 2's calibration must degrade to a no-op rather than a bad estimate. Criterion 8 covers this. **Heterogeneous nodes.** A single configured budget assumes uniform node sizing, so clusters with mixed node sizes will be planned against the wrong number until Option B lands. Worth calling out in the setting's documentation. **Interaction with the buffer factor.** `global_buffer_factor` is recomputed hourly from live data (`ee/app/models/search/zoekt/index.rb:126`), so `scaled_size` (`ee/app/services/search/zoekt/planning_service.rb:249`) already moves under the planner; a memory estimate derived from a scaled size would compound the two ratios and inherit that volatility. 1a avoids it by dropping the second multiplication — not by changing which disk figure is the base, and notably not by using `used_storage_bytes`, which is unavailable at plan time on the `:create` path (`ee/app/models/search/zoekt/index.rb:252`). ## References - Originating request for help (internal link; not accessible to community readers): https://gitlab.com/gitlab-com/request-for-help/-/issues/5304 - Zoekt memory architecture and sizing guidance: https://docs.gitlab.com/integration/zoekt/#memory-architecture Every code citation in this issue is inline at the point it is used. The files involved, as a pointer list: - Planner — placement, packing, node exhaustion, plan partitioning: `ee/app/services/search/zoekt/planning_service.rb:186`; index cap and its break-out, `ee/app/models/search/zoekt.rb:8` and `ee/app/services/search/zoekt/planning_service.rb:144`. - Node capacity and disk watermarks: `ee/app/models/search/zoekt/node.rb:285`. - Memory telemetry, collected and persisted but display-only: `ee/lib/api/internal/search/zoekt.rb:24`, `ee/app/services/search/zoekt/info_service.rb:286`. - Search-routing exclusion on mmap, not RSS: `ee/lib/search/zoekt/process_health.rb:20`; SaaS-gated eviction, `ee/app/services/search/zoekt/scheduling_service.rb:181`. - Buffer factor (the pattern 1a mirrors in shape) and the settings hash: `ee/app/models/search/zoekt/index.rb:165`, `ee/app/models/search/zoekt/settings.rb:34`. - Self-managed rollout path, plan log, and failed-plan provisioning: `ee/app/workers/search/zoekt/rollout_worker.rb:30`, `ee/app/services/search/zoekt/rollout_service.rb:44`, `ee/app/services/search/zoekt/provisioning_service.rb:159`.