### Summary `Search::Zoekt::PlanningService` plans a new replica without knowing which nodes the namespace's existing replicas already occupy. It can therefore plan an index on a node that already holds an index for the same enabled namespace, violating the unique index `u_zoekt_indices_zoekt_enabled_namespace_id_and_zoekt_node_id` on `zoekt_indices (zoekt_enabled_namespace_id, zoekt_node_id)`. That index does not include `zoekt_replica_id`, so the rule is one index per node **per namespace**, not per replica. Replicas of a namespace must occupy disjoint node sets. This is dormant while `zoekt_default_number_of_replicas` is `1`. It becomes reachable as soon as that setting is raised. ### Steps to reproduce 1. Take an enabled namespace with one provisioned replica whose index sits on the node with the most unclaimed space. 1. Set `number_of_replicas_override` to `2` on that namespace, or raise `zoekt_default_number_of_replicas`. 1. Run `Search::Zoekt::PlanningService.plan` for it. 1. Inspect the returned replica's indices for a `node_id` the namespace already uses. ### What is the current *bug* behavior? **Why it is dormant today.** `SelectionService` picks namespaces via `EnabledNamespace.with_mismatched_replicas`, which is `COUNT(replicas) != COALESCE(number_of_replicas_override, zoekt_default_number_of_replicas)`. While the default is `1`, a namespace already holding one replica is not mismatched and is never handed to the planner: ``` namespace has replicas=1 indices=1 desired 1 -> mismatched? false desired 2 -> mismatched? true SelectionService returns this namespace? true ``` Raising the default to `2` makes every already-provisioned namespace mismatched, selected, and planned — with indices already in the database. **The mechanism.** `Plan#generate` plans only the delta, `replicas_to_create = desired_replicas_count - existing_replicas_count`, which for a 1 to 2 transition is a single new replica. The set of nodes excluded from selection is built only from replicas planned in the current run — `replica_plans` plus the replica being packed. `replica_plans` starts empty on every run and is never seeded from the database, so the namespace's existing indices are invisible to it. `find_best_node` then returns the first entry in `nodes`, which `format_nodes` sorts by unclaimed space descending, and nothing removes a node the namespace already occupies. **How often it fires.** It is conditional: it collides whenever the existing index sits on a node that is still among the emptiest. Two runs with identical setup, differing only in which node holds the existing index: ``` A existing index on node 1780 (19.0 GiB, still emptiest) -> new replica plans [1780] -> COLLISION = true B existing index on node 1779 (4.0 GiB, no longer emptiest) -> new replica plans [1778] -> COLLISION = false ``` The condition is common rather than rare. The existing replica was itself placed by the same emptiest-first rule, so it started on a top node, and its own reservation pushes that node down the ordering only by its own size. A small namespace reserving a few GB on a large node stays near the top. Large namespaces are better protected because their reservation is bigger. Namespaces at `0` replicas are unaffected. When both replicas are planned in the same run, `replica_plans` accumulates between them and the nodes come out disjoint. Verified end to end — a fresh namespace with a desired count of `2` produced two replicas on distinct nodes with no errors. **How it presents.** The plan is returned clean, with an empty `errors` array, and lands in the `create` bucket. `ProvisioningService`'s `Index.insert_all!` then raises `PG::UniqueViolation`, which `process_replica` rescues as a generic `StandardError` and aggregates into the plan errors. The surrounding transaction rolls back, so no replica and no indices are created, and `update_enabled_namespace` stamps `last_rollout_failed_at`, which blocks retries through `EnabledNamespace.with_rollout_allowed` until the retry interval elapses. Externally the second replica silently never appears. ### What is the expected *correct* behavior? `PlanningService.plan` should never return a plan placing an index on a node that already holds an index for the same enabled namespace, whether that index was planned in the current run or already exists in the database. Raising `zoekt_default_number_of_replicas` should provision second replicas onto nodes disjoint from the existing ones, or record an actionable error when no such node has capacity. ### Relevant logs and/or screenshots Planning a second replica for a namespace that already has one provisioned: ``` existing replicas in DB: 1, on nodes [1767] action=create replicas_planned=1 new replica would use nodes: [1767] COLLIDES WITH EXISTING: true errors=[] ``` The empty `errors` array is the important part — the plan reports success and the failure only surfaces at provisioning. ### Possible fixes 1. Seed the exclusion set from the database as well as the current run, for example by adding `enabled_namespace.indices.pluck(:zoekt_node_id)` to the node ids already excluded. Small change, and it makes the planner consistent with the constraint. 1. Treat a namespace whose desired replica count has increased as a full re-plan rather than a delta, so all replicas are planned together and the existing ones are visible. Larger behavioural change. 1. Add a guard in `ProvisioningService` filtering out any planned index whose node already holds one for the namespace, so an inconsistent plan degrades rather than raising. A related issue is already open for a different trigger in the same area, where the planner reuses a node **within** a single replica. That one is reachable today; this one requires the replica count to be raised first.