Zoekt PlanningService can plan two indices on the same node for one namespace

Summary

Search::Zoekt::PlanningService can produce a plan that assigns two indices to the same node for the same enabled namespace. This violates the unique index u_zoekt_indices_zoekt_enabled_namespace_id_and_zoekt_node_id on zoekt_indices (zoekt_enabled_namespace_id, zoekt_node_id). If such a plan is provisioned, ProvisioningService#process_indices!'s Index.insert_all! raises PG::UniqueViolation.

Steps to reproduce

Run Search::Zoekt::PlanningService.plan for an enabled namespace whose total required storage exceeds the emptiest node's unclaimed space, where the first index leaves a remainder large enough to hold at least one later project. Inspect the returned plan's replicas for a repeated node_id.

This reproduces at plan time and requires no provisioning.

Observed on production: a dry-run plan for namespace 8338830 (enabled namespace 2159361) produced 10 indices across only 9 distinct nodes. Node 1000198 (gitlab-gitlab-zoekt-31) received two:

  • index 1: range (..49853664), required_storage_bytes 71,244,546,477, max_storage_bytes 71,859,053,179
  • index 4: range (79422119..80421166), required_storage_bytes 389,945,383, max_storage_bytes 614,506,701

71,859,053,179 minus 71,244,546,477 equals 614,506,702, which is exactly index 4's max_storage_bytes. simulate_index sets max_storage_bytes to the node's current unclaimed space, so index 4 was sized against node 1000198's leftover and no other node's, confirming it reused that node.

The immediately preceding index (index 3, on node 2000230) was full — 24,921,315,933 of 24,944,370,808 bytes, leaving 23,054,875 bytes — so the next project could not extend it and a new index was required. find_best_node then returned 1000198 because it was first in the array and not exhausted.

That particular plan failed earlier with index_limit_exceeded (MAX_INDICES_PER_REPLICA is 10), so it never reached provisioning. The failure is incidental: a plan with no errors at all can carry the same duplicate and does hit the unique violation, as shown below.

What is the current bug behavior?

In ee/app/services/search/zoekt/planning_service.rb, assign_project_to_index (line 205) only retires a node into @exhausted_node_ids when last_index is present:

@exhausted_node_ids.add(node[:id]) if last_index # If this is the first project of the replica then skip

When the first index of a replica is created, replica_indices is still empty, so last_index is nil and the guard is false — that node is never added to @exhausted_node_ids. Every subsequent index retires its node immediately, but the first index's node stays eligible for the entire packing loop and is only retired at the end, by add_replica_plan (line 238).

find_best_node (line 186) does a first-match scan:

nodes.find { |n| !@exhausted_node_ids.member?(n[:id]) && (n[:unclaimed_storage_bytes] >= scaled_size(stats)) }

nodes is sorted by unclaimed space descending in format_nodes and never re-sorted, so the emptiest node is permanently nodes[0] and is checked first on every lookup. Because that node is never exhausted early, it can be selected again for a later index in the same replica.

The comment on add_replica_plan at line 237 states the exhaustion set exists "to ensure indices of different replicas of a namespace does not get the same node." The same constraint needs to hold within a single replica, and the if last_index exemption on line 205 defeats it there.

Any namespace whose first index leaves a usable remainder on its node, and which later needs a new index for a project small enough to fit that remainder, can produce this plan.

The duplicate is not limited to failing plans

The production plan that first surfaced this bug ended in index_limit_exceeded, which might suggest the duplicate node id only shows up once a plan is already failing. That is not the case. A three-node scenario run against the unfixed planner produced a plan that landed in the create bucket with an empty errors array, and its replica still contained a repeated node id:

plan buckets: create=1 failures=0 unchanged=0
errors: []
indices=3 node_ids=[244, 245, 244] distinct=2

Passing that same plan to ProvisioningService.execute then produced a genuine failure:

error: PG::UniqueViolation: ERROR: duplicate key value violates unique constraint
       "u_zoekt_indices_zoekt_enabled_namespace_id_and_zoekt_node_id"
replicas now: 0
indices now : 0
enabled_namespace.last_rollout_failed_at: 2026-09-04 10:55:38 UTC

Why most rollouts are unaffected: the duplicate requires at least three indices, and cannot occur with fewer. Index 1 fills the first node and leaves a remainder. The project that overflows index 1 is by definition larger than that remainder, so find_best_node skips the first node and index 2 always lands elsewhere. Index 3 is the first opportunity, because the project that overflows index 2 must exceed the second node's remainder but may still fit inside the first node's remainder. The first node is checked before all others, since nodes is sorted by unclaimed space descending and never re-sorted, and it is never retired. Any namespace that fits in one or two indices is therefore immune, which is the large majority.

How it presents when it does occur: ProvisioningService#process_replica rescues StandardError, so the violation is aggregated into the plan errors rather than raised. The surrounding transaction rolls back, leaving no replica and no indices, and update_enabled_namespace stamps last_rollout_failed_at. That value then blocks retries through EnabledNamespace.with_rollout_allowed until the retry interval elapses. Externally this looks like a large namespace that will not provision, rather than a crash.

It would be worth searching production provisioning logs for the string duplicate key value violates unique constraint "u_zoekt_indices_zoekt_enabled_namespace_id_and_zoekt_node_id" to find how often this has actually fired.

What is the expected correct behavior?

A single replica's indices should never land on the same node_id. PlanningService.plan should never return a plan with a repeated node_id within one replica's indices, regardless of which index in packing order was the first to use that node.

Relevant logs and/or screenshots

Dry-run plan for enabled namespace 2159361 (namespace 8338830), relevant indices:

index 1: range (..49853664), node_id 1000198, required_storage_bytes 71244546477, max_storage_bytes 71859053179
index 3: range (...),        node_id 2000230, required_storage_bytes 24921315933, max_storage_bytes 24944370808
index 4: range (79422119..80421166), node_id 1000198, required_storage_bytes 389945383, max_storage_bytes 614506701

Plan failed with index_limit_exceeded before reaching provisioning, so no PG::UniqueViolation was raised in this instance.

Possible fixes

  1. Remove the if last_index guard on line 205 so the first index's node is retired like any other node once it is used. Minimal change, but it forfeits the first node's remaining capacity for the rest of the replica's packing, which is presumably what the exemption was meant to preserve.
  2. Make find_best_node skip nodes that already hold an index for the current replica, for example by checking replica_indices for the node id before returning it. Keeps the remaining capacity usable for the current index rather than discarding it, but is a larger change to the selection logic.
  3. Let the current index keep filling its node's remainder before a new index is opened, addressing the same capacity-waste concern the exemption was targeting without ever reusing a node for two indices of the same replica.
Edited by Ravi Kumar