# Memory-aware placement for the Zoekt planning service ## Goal Give `Search::Zoekt::PlanningService` a memory dimension so it can force a hot namespace to split across nodes when disk alone would keep it on one. Today the planner reasons only about bytes on disk, and the resource that actually saturates a Zoekt webserver in production is resident memory. The originating evidence is a self-managed customer escalation: two nodes at ~27% and ~6% disk used, carrying ~136 GiB and ~5 GiB webserver resident memory respectively — a ~27x memory imbalance that no disk watermark can ever detect, because `storage_percent_used` is `used_bytes / total_bytes` (`ee/app/models/search/zoekt/node.rb:285`) and the watermark limits are 0.6 / 0.75 / 0.85 (`ee/app/models/search/zoekt/node.rb:13`). The full problem statement, options analysis, scope boundaries, risk register and twelve acceptance criteria live in the upstream issue and are **not** duplicated here: - https://gitlab.com/gitlab-org/gitlab/-/work_items/627279 **This is not a single-customer problem.** The same failure class was worked by hand on GitLab.com a week later: a large namespace could not be indexed for lack of placement room, and the remedy was a manual Rails-console force-split plus two new nodes (https://gitlab.com/gitlab-com/request-for-help/-/work_items/5344). That namespace — `8338830` — is the same one whose dry-run plan produced 10 indices across only 9 distinct nodes in https://gitlab.com/gitlab-org/gitlab/-/work_items/627730. So the forced split this epic proposes as a planner primitive has already been performed manually, twice, on two different deployments: once by a self-managed operator shrinking PVCs to lie to the planner, once by an engineer partitioning a namespace from a console. ## Why this is tractable now **The telemetry already exists.** Per-node webserver `rss_bytes` and `shards_loaded` arrive on every heartbeat (`ee/lib/api/internal/search/zoekt.rb:24`, `ee/lib/api/internal/search/zoekt.rb:90`), are persisted into `zoekt_nodes.metadata` (`ee/app/models/search/zoekt/node.rb:336`), and are validated by a JSON schema that names webserver RSS explicitly (`ee/app/validators/json_schemas/zoekt_node_metadata.json:68`). Their only consumer outside specs is human-readable CLI output in `gitlab:zoekt:info` (`ee/app/services/search/zoekt/info_service.rb:286`). **The gap is consumption, not collection.** No scheduler, planner, eviction path or routing decision reads resident memory. That is what makes this a bounded planner change rather than a telemetry project. ## Done when 1. A namespace whose estimated resident cost exceeds a configured per-index memory ceiling is split across **distinct** nodes, proven by a planner unit test asserting every index in the plan has a different `node_id` — not merely that more than one node appears. 2. With the per-node memory budget setting unset, the planner emits a byte-for-byte identical plan to today's. The feature is inert until configured. 3. A memory ceiling that cannot be satisfied on the available nodes falls back to the disk-only decision and logs a warning, and does **not** accumulate an error into `plan[:errors]` (`ee/app/services/search/zoekt/planning_service.rb:22`). Un-provisioning the hot namespace is strictly worse than today's behaviour. 4. Memory-driven node exhaustion is attributable — distinguishable in the diagnostic from disk-driven exhaustion by `type` or `details`. 5. The unconditional rollout plan log (`ee/app/services/search/zoekt/rollout_service.rb:44`) carries the estimated memory cost per index and the per-node budget before and after. 6. No `Gitlab::Saas.feature_available?` call exists anywhere in the `RolloutWorker` → `RolloutService` → `PlanningService` chain, enforced by a regression test. The chain is ungated today and must stay so — this was reported by a self-managed customer, and `eviction` is already SaaS-gated at `ee/app/services/search/zoekt/scheduling_service.rb:181`. 7. Documentation states both new settings, their defaults, and plainly that on a cluster with N nodes a namespace can be split at most N ways. The upstream issue's acceptance criteria are authoritative where they are more specific than the above. ## Scope In: - A per-index estimated memory cost, `repository_size * memory_ratio`, from the same `project.statistics` figure the planner already reads (`ee/app/services/search/zoekt/planning_service.rb:141`). - A per-node memory budget carried into the planner's node model beside `unclaimed_storage_bytes` (`ee/app/services/search/zoekt/planning_service.rb:45`). - A maximum-memory-per-index ceiling, plus the node-selection reordering needed so a memory-driven rollover lands on a **different** node. - Graceful degradation on unsatisfiable memory constraints. - Memory fields in the plan log; estimated-vs-observed in `gitlab:zoekt:info`. - Identical behaviour on self-managed and SaaS. Out: - Memory-driven eviction or rebalancing of already-placed indices. - Repartitioning an existing namespace without a reindex. - Node-side changes to the Zoekt indexer or webserver. The cgroup-reported memory limit is a deliberate follow-up. - Changing `MAX_INDICES_PER_REPLICA` from 10, or relaxing the one-index-per-node intra-replica constraint (`ee/app/services/search/zoekt/planning_service.rb:205`). - A memory-denominated watermark predicate. Worthwhile, but it touches node-level predicates rather than the planner. - Per-namespace or query-rate-weighted ratios, and automatic self-tuning. ## Hard constraints - **The node-selection ordering is the load-bearing part.** `assign_to_node` calls `find_best_node` *before* the packing decision (`ee/app/services/search/zoekt/planning_service.rb:175` vs `:192`), and today that is harmless only by coincidence: `max_storage_bytes` is seeded from the node's unclaimed disk at `:219` and decremented in lockstep at `:209`, so a disk-driven rollover finds the node already empty. A memory ceiling deliberately breaks that coupling — when it binds, the node still has abundant disk, and the naive implementation puts the second index on the **same node**: zero memory relief while appearing to work. Both changes are required — add the node to `@exhausted_node_ids` at the moment the memory ceiling is identified as the reason for the roll, and re-select the candidate node *after* the packing test. - **Do not derive the memory estimate from `used_storage_bytes`.** It is populated only by `refresh_used_storage_bytes` from `zoekt_repositories.sum(:size_bytes)` (`ee/app/models/search/zoekt/index.rb:252`) and falls back to `DEFAULT_USED_STORAGE_BYTES = 1.kilobyte` (`ee/app/models/search/zoekt/index.rb:10`). On the `:create` path the planner invents indices with no repository rows, so it is absent or 1 KiB precisely when the forced-split decision must be made. - **`memory_ratio` is not a disk-to-disk ratio and must not be multiplied by `buffer_factor`.** It maps raw repository bytes to resident bytes. The buffer factor is recomputed hourly from live data (`ee/app/models/search/zoekt/index.rb:126`), so compounding the two inherits that volatility. - **Under-estimating is the dangerous direction** — the planner believes it has headroom it does not have and reproduces the reported failure silently. Default conservatively. ## Prior art re-examined Two closed issues bear directly on this work and were checked rather than assumed. Both are authored by the same team and labelled complete, and reading them changes the plan: **Anti-affinity was already requested, and shipped only partially.** https://gitlab.com/gitlab-org/gitlab/-/work_items/593209 ("Ensure namespace data has replica redundancy across multiple nodes") asked, in its section 1, to "ensure that when creating replicas for a namespace, the indices are placed on distinct nodes (anti-affinity)". It is closed `workflow::complete`. The guarantee holds when a namespace's replicas are planned **in the same run** — verified end to end in https://gitlab.com/gitlab-org/gitlab/-/work_items/627737, where a fresh namespace with a desired count of 2 produced two replicas on distinct nodes. It does **not** hold on the incremental path, because `replica_plans` starts empty on every run and is never seeded from the database, so a 1-to-2 replica transition cannot see where the existing index sits. The consequence for this epic: memory-driven splitting depends on the planner being able to keep one namespace's indices on distinct nodes, and that property is currently guaranteed only for the all-at-once case. Do not treat 593209 as having delivered the precondition. **The estimate-error risk has a documented precedent.** https://gitlab.com/gitlab-org/gitlab/-/work_items/508836 ("Improve zoekt namespace assignment service", closed, `missed-deliverable`, `missed:17.7`, `missed:17.8`) recorded that "the namespace root storage statistics and project statistics may or may not be accurate to actual used bytes in a zoekt index", and proposed using `last_used_storage_bytes` where available instead. This matters because the memory estimate in this epic is `repository_size * memory_ratio`, and `repository_size` is exactly the project-statistics figure 508836 flagged as an unreliable proxy for real index bytes. The proposal still prefers it — `used_storage_bytes` is absent or 1 KiB on the `:create` path, which is precisely when the forced-split decision is made — but the inherited error is a known quantity, not a new risk, and it is one more argument for defaulting the ratio conservatively and for shipping Step 2's estimated-versus-observed comparison close behind Step 1. ## Existing telemetry and monitoring, verified Checked directly rather than assumed, because Step 1's ratio needs a calibration source and because the cost of Option B turned out to be mis-stated. **The node already bridges its Prometheus registry into the heartbeat.** Verified in `gitlab-org/gitlab-zoekt-indexer` at `76b0f9c4943f9ffa9ada00ccf539e2b4a4b86c08` (`VERSION` 1.19.1). The heartbeat and the task-request are **one HTTP request**: `buildRequestPayload` (`internal/task_request/task_request.go:182-229`) builds a single flat `map[string]interface{}` carrying the `node.*` fields, `disk.all` / `disk.free` / `disk.used` / `disk.indexed`, and the `process_health` block. `ProcessMetrics` (`internal/shared/processhealth/processhealth.go:38-50`) is populated by `collectFromMFs` (`:92`) switching on metric names from `prometheus.DefaultGatherer.Gather()` (`:85`) — `process_resident_memory_bytes` becomes `rss_bytes`, `zoekt_shards_loaded` becomes `shards_loaded`, and so on. So the `rss_bytes` this epic proposes calibrating against **is already a Prometheus metric**, and the payload is an untyped map. Adding a memory-capacity field is one `case` in that switch plus one struct field. `gopsutil/v4` is already a direct dependency (`go.mod:18`) and already used for disk (`internal/mode/webserver/webserver.go:51`, `:562-567`). Version fencing needs no new mechanism: the payload already carries `node.version` (`:194`) and an integer `node.schema_version`, currently `2602` (`:34`, `:195`). What is genuinely absent: a grep for `cgroup`, `memory.max`, `memory.limit_in_bytes`, `MemTotal`, `MemAvailable`, `/proc/meminfo`, `GOMEMLIMIT` and `SetMemoryLimit` across all non-test Go files returns **nothing**. The container memory *limit* — which is what Kubernetes enforces and what the reporting customer was constrained by — is not read anywhere today. **Consequence for the options table:** Option B's cost is over-stated there. It needs no new telemetry plumbing, no new dependency, and no new version fence. The open question is *which number* to report (host `MemAvailable` is misleading inside a container; the cgroup limit is the enforced one and is the part that does not exist yet). Recommendation order is unchanged — Option A ships on every deployment on day one and makes the constraint auditable — but B belongs in the same milestone rather than a distant follow-up, and Option A's setting documentation should state that a reported value will supersede it. **Production monitoring already measures this, and only alerts after the fact.** Verified in `gitlab-com/runbooks`: - `mimir-rules/gitlab-gprd/search/zoekt.yml` (and its `gitlab-gstg` twin) define exactly two alerts plus one recording rule. The recording rule `gitlab:zoekt_gke_oom_events` joins `kube_pod_container_status_last_terminated_reason{reason="OOMKilled"}` against restart deltas; `ZoektContainersRestartsDueToOOM` (severity `s3`, `user_impacting: "yes"`) fires on it. The other alert, `ZoektNodesOffline`, is liveness, not memory. - `docs/zoekt/README.md` carries a ready-made Thanos query, `sum(process_resident_memory_bytes{env="gprd", container=~"zoekt.*"}) by (container, pod)`, and an OOM companion query. This is the strongest available argument for the epic: **the only memory signal in production fires after a node has already been OOM-killed.** There is no saturation metric, no alert, and no watermark on memory *imbalance* — every watermark, SLO and runbook remedy in the service is disk-denominated. Placement is the earliest point at which the imbalance is representable at all. **Calibration data exists, with a caveat that bounds the design.** `process_resident_memory_bytes` is retained in Thanos, so the estimated-versus- observed comparison in Step 2 has real history to work against, and the ratio's default can be anchored rather than guessed. The caveat: the documented memory architecture states resident memory grows with the **active working set**, i.e. with query volume and not index size alone. A single global memory-per-byte constant will therefore fit the hottest and coldest namespaces badly at the same time — which is exactly why Step 1 states the global ratio as a deliberate first increment and why Step 2 must not lag far behind it. Prefer joining `repository_size` from the Rails database over stacking a second observed ratio on top of the storage buffer factor. ## GitLab.com's own configuration is the strongest argument here Verified in `gitlab-com/gl-infra/k8s-workloads/gitlab-com`, `releases/gitlab/values/gprd.yaml.gotmpl`. Three facts from that one file: **1. GitLab.com avoids this failure by brute force.** gprd runs `replicas: 38` with a webserver `requests.memory: 128Gi` and `limits.memory: 256Gi` on a `generic-mem` node pool. With 38 nodes, a hot namespace exceeds a single node's disk and the disk-only planner splits it as a side effect. The reporting customer ran **two** nodes with 1 TiB volumes, so nothing ever triggered a split. The current planner is only adequate at node counts high enough to make the disk constraint bind — it has no direct expression of the thing being managed. **2. Memory saturation alerting is explicitly switched off for Zoekt.** ```yaml # Memory usage above requests is intentional for zoekt (mmap'd index shards), and stays # well within limits, so exclude it from the request-based saturation point. statefulSet: podLabels: gitlab.com/exclude_saturation_point: kube_container_memory_request ``` **3. Capacity planning already flagged it, and the resolution was to silence the metric rather than manage the resource.** `gitlab-com/gl-infra/capacity-planning-trackers/gitlab-com` work item 2667 ("Capacity warning for zoekt: kube_container_memory_request", closed) records a **soft SLO violation forecast dated 2026-09-05** and, in its own component header, `Horizontally scalable: False`. Its stated reason for dismissal is technically sound — the cAdvisor working-set figure "includes filesystem cache pages that are not necessarily attributable to the application inside the cgroup, and are permitted to be evicted instead of being OOM killed." The exclusion is defensible for *alerting* and simultaneously makes the case for this epic: the only automated memory signal Zoekt had was measuring the wrong thing, so it was correctly turned off, and nothing replaced it. `Horizontally scalable: False` is precisely the property memory-aware placement changes — a namespace that can be split across nodes on a memory criterion is horizontally scalable on memory. Cite work item 2667 when justifying this work to infrastructure; it is their own tracker reaching the same conclusion from the opposite direction. **Corollary for Option B's open question — report the cgroup limit, not host RAM.** On gprd the enforced number is `limits.memory: 256Gi`, and the customer's constraint was likewise a per-pod limit (their pods exceeded a 65 GiB scheduling SLO). Host `MemAvailable` would report the node's physical RAM and badly overstate a pod's budget; work item 2667 documents that the working-set figure is unreliable in the other direction. The cgroup limit is the only figure that matches what Kubernetes actually enforces, and it is the one value the indexer does not read today. ## An unpropagated OOM mitigation, found while checking the above Not in scope for this epic, but it is a live divergence worth someone's attention and it corroborates the failure mode. `gitlab-com/gl-infra/k8s-workloads/gitlab-com!5241` ("Mitigate Zoekt webserver OOM crashloops in gstg", merged 2026-03-11) applied two P0 mitigations to **gstg** only: - `ZOEKT_DOCMATCHTREE_CACHE` 64 to 0 — "eliminates Go heap memory used for caching `docMatchTree` structures across many index shards" - webserver memory 4Gi/8Gi to 16Gi/32Gi — "headroom for mmap'd shard files from the `gitlab-org` group index" It closed https://gitlab.com/gitlab-org/gitlab/-/work_items/593204 and https://gitlab.com/gitlab-org/gitlab/-/work_items/593205 under parent epic https://gitlab.com/groups/gitlab-org/-/epics/21301. Current state: `gstg.yaml.gotmpl` has the cache commented out (`# ZOEKT_DOCMATCHTREE_CACHE: 128`), while **gprd still sets `ZOEKT_DOCMATCHTREE_CACHE: 64`**. So a P0 mitigation for webserver OOM crashloops has been in place on staging for six months and was never propagated to production, where the same 38-node fleet runs 256Gi limits. That is either deliberate — gprd's much larger limits may absorb it — or an oversight. It needs an infrastructure answer rather than a guess, and it is filed here only so the observation is not lost. ## Related - Upstream issue (authoritative spec): https://gitlab.com/gitlab-org/gitlab/-/work_items/627279 - Originating customer escalation (internal): https://gitlab.com/gitlab-com/request-for-help/-/work_items/5304 - Two independent planner correctness bugs in the same service, found while this was being written. Both are node-selection defects in `find_best_node` / `@exhausted_node_ids` — the same code this epic modifies — so they should be read before implementation, and may be worth fixing first: - https://gitlab.com/gitlab-org/gitlab/-/work_items/627730 — the planner can plan two indices on the same node for one namespace, violating `u_zoekt_indices_zoekt_enabled_namespace_id_and_zoekt_node_id`. - https://gitlab.com/gitlab-org/gitlab/-/work_items/627737 — planning a new replica ignores the nodes existing replicas already occupy. Dormant while `zoekt_default_number_of_replicas` is 1. - Zoekt memory architecture and sizing guidance: https://docs.gitlab.com/integration/zoekt/#memory-architecture