### Summary `Search::Zoekt::PlanningService` can produce a plan that assigns two indices to the same node for the same enabled namespace. This violates the unique index `u_zoekt_indices_zoekt_enabled_namespace_id_and_zoekt_node_id` on `zoekt_indices (zoekt_enabled_namespace_id, zoekt_node_id)`. If such a plan is provisioned, `ProvisioningService#process_indices!`'s `Index.insert_all!` raises `PG::UniqueViolation`. ### Steps to reproduce Run `Search::Zoekt::PlanningService.plan` for an enabled namespace whose total required storage exceeds the emptiest node's unclaimed space, where the first index leaves a remainder large enough to hold at least one later project. Inspect the returned plan's `replicas` for a repeated `node_id`. This reproduces at plan time and requires no provisioning. Observed on production: a dry-run plan for namespace `8338830` (enabled namespace `2159361`) produced 10 indices across only 9 distinct nodes. Node `1000198` (`gitlab-gitlab-zoekt-31`) received two: - index 1: range `(..49853664)`, `required_storage_bytes` 71,244,546,477, `max_storage_bytes` 71,859,053,179 - index 4: range `(79422119..80421166)`, `required_storage_bytes` 389,945,383, `max_storage_bytes` 614,506,701 71,859,053,179 minus 71,244,546,477 equals 614,506,702, which is exactly index 4's `max_storage_bytes`. `simulate_index` sets `max_storage_bytes` to the node's current unclaimed space, so index 4 was sized against node `1000198`'s leftover and no other node's, confirming it reused that node. The immediately preceding index (index 3, on node `2000230`) was full — 24,921,315,933 of 24,944,370,808 bytes, leaving 23,054,875 bytes — so the next project could not extend it and a new index was required. `find_best_node` then returned `1000198` because it was first in the array and not exhausted. That particular plan failed earlier with `index_limit_exceeded` (`MAX_INDICES_PER_REPLICA` is 10), so it never reached provisioning. The failure is incidental: a plan with no errors at all can carry the same duplicate and does hit the unique violation, as shown below. ### What is the current *bug* behavior? In `ee/app/services/search/zoekt/planning_service.rb`, `assign_project_to_index` (line 205) only retires a node into `@exhausted_node_ids` when `last_index` is present: ```ruby @exhausted_node_ids.add(node[:id]) if last_index # If this is the first project of the replica then skip ``` When the first index of a replica is created, `replica_indices` is still empty, so `last_index` is `nil` and the guard is false — that node is never added to `@exhausted_node_ids`. Every subsequent index retires its node immediately, but the first index's node stays eligible for the entire packing loop and is only retired at the end, by `add_replica_plan` (line 238). `find_best_node` (line 186) does a first-match scan: ```ruby nodes.find { |n| !@exhausted_node_ids.member?(n[:id]) && (n[:unclaimed_storage_bytes] >= scaled_size(stats)) } ``` `nodes` is sorted by unclaimed space descending in `format_nodes` and never re-sorted, so the emptiest node is permanently `nodes[0]` and is checked first on every lookup. Because that node is never exhausted early, it can be selected again for a later index in the same replica. The comment on `add_replica_plan` at line 237 states the exhaustion set exists "to ensure indices of different replicas of a namespace does not get the same node." The same constraint needs to hold within a single replica, and the `if last_index` exemption on line 205 defeats it there. Any namespace whose first index leaves a usable remainder on its node, and which later needs a new index for a project small enough to fit that remainder, can produce this plan. ### The duplicate is not limited to failing plans The production plan that first surfaced this bug ended in `index_limit_exceeded`, which might suggest the duplicate node id only shows up once a plan is already failing. That is not the case. A three-node scenario run against the unfixed planner produced a plan that landed in the `create` bucket with an empty `errors` array, and its replica still contained a repeated node id: ``` plan buckets: create=1 failures=0 unchanged=0 errors: [] indices=3 node_ids=[244, 245, 244] distinct=2 ``` Passing that same plan to `ProvisioningService.execute` then produced a genuine failure: ``` error: PG::UniqueViolation: ERROR: duplicate key value violates unique constraint "u_zoekt_indices_zoekt_enabled_namespace_id_and_zoekt_node_id" replicas now: 0 indices now : 0 enabled_namespace.last_rollout_failed_at: 2026-09-04 10:55:38 UTC ``` Why most rollouts are unaffected: the duplicate requires at least three indices, and cannot occur with fewer. Index 1 fills the first node and leaves a remainder. The project that overflows index 1 is by definition larger than that remainder, so `find_best_node` skips the first node and index 2 always lands elsewhere. Index 3 is the first opportunity, because the project that overflows index 2 must exceed the second node's remainder but may still fit inside the first node's remainder. The first node is checked before all others, since `nodes` is sorted by unclaimed space descending and never re-sorted, and it is never retired. Any namespace that fits in one or two indices is therefore immune, which is the large majority. How it presents when it does occur: `ProvisioningService#process_replica` rescues `StandardError`, so the violation is aggregated into the plan errors rather than raised. The surrounding transaction rolls back, leaving no replica and no indices, and `update_enabled_namespace` stamps `last_rollout_failed_at`. That value then blocks retries through `EnabledNamespace.with_rollout_allowed` until the retry interval elapses. Externally this looks like a large namespace that will not provision, rather than a crash. It would be worth searching production provisioning logs for the string `duplicate key value violates unique constraint "u_zoekt_indices_zoekt_enabled_namespace_id_and_zoekt_node_id"` to find how often this has actually fired. ### What is the expected *correct* behavior? A single replica's indices should never land on the same `node_id`. `PlanningService.plan` should never return a plan with a repeated `node_id` within one replica's indices, regardless of which index in packing order was the first to use that node. ### Relevant logs and/or screenshots Dry-run plan for enabled namespace `2159361` (namespace `8338830`), relevant indices: ``` index 1: range (..49853664), node_id 1000198, required_storage_bytes 71244546477, max_storage_bytes 71859053179 index 3: range (...), node_id 2000230, required_storage_bytes 24921315933, max_storage_bytes 24944370808 index 4: range (79422119..80421166), node_id 1000198, required_storage_bytes 389945383, max_storage_bytes 614506701 ``` Plan failed with `index_limit_exceeded` before reaching provisioning, so no `PG::UniqueViolation` was raised in this instance. ### Possible fixes 1. Remove the `if last_index` guard on line 205 so the first index's node is retired like any other node once it is used. Minimal change, but it forfeits the first node's remaining capacity for the rest of the replica's packing, which is presumably what the exemption was meant to preserve. 1. Make `find_best_node` skip nodes that already hold an index for the current replica, for example by checking `replica_indices` for the node id before returning it. Keeps the remaining capacity usable for the current index rather than discarding it, but is a larger change to the selection logic. 1. Let the current index keep filling its node's remainder before a new index is opened, addressing the same capacity-waste concern the exemption was targeting without ever reusing a node for two indices of the same replica.