Zoekt PlanningService ignores existing indices when adding a replica

Summary

Search::Zoekt::PlanningService plans a new replica without knowing which nodes the namespace's existing replicas already occupy. It can therefore plan an index on a node that already holds an index for the same enabled namespace, violating the unique index u_zoekt_indices_zoekt_enabled_namespace_id_and_zoekt_node_id on zoekt_indices (zoekt_enabled_namespace_id, zoekt_node_id).

That index does not include zoekt_replica_id, so the rule is one index per node per namespace, not per replica. Replicas of a namespace must occupy disjoint node sets.

This is dormant while zoekt_default_number_of_replicas is 1. It becomes reachable as soon as that setting is raised.

Steps to reproduce

  1. Take an enabled namespace with one provisioned replica whose index sits on the node with the most unclaimed space.
  2. Set number_of_replicas_override to 2 on that namespace, or raise zoekt_default_number_of_replicas.
  3. Run Search::Zoekt::PlanningService.plan for it.
  4. Inspect the returned replica's indices for a node_id the namespace already uses.

What is the current bug behavior?

Why it is dormant today. SelectionService picks namespaces via EnabledNamespace.with_mismatched_replicas, which is COUNT(replicas) != COALESCE(number_of_replicas_override, zoekt_default_number_of_replicas). While the default is 1, a namespace already holding one replica is not mismatched and is never handed to the planner:

namespace has replicas=1 indices=1
desired 1 -> mismatched? false
desired 2 -> mismatched? true
SelectionService returns this namespace? true

Raising the default to 2 makes every already-provisioned namespace mismatched, selected, and planned — with indices already in the database.

The mechanism. Plan#generate plans only the delta, replicas_to_create = desired_replicas_count - existing_replicas_count, which for a 1 to 2 transition is a single new replica. The set of nodes excluded from selection is built only from replicas planned in the current run — replica_plans plus the replica being packed. replica_plans starts empty on every run and is never seeded from the database, so the namespace's existing indices are invisible to it. find_best_node then returns the first entry in nodes, which format_nodes sorts by unclaimed space descending, and nothing removes a node the namespace already occupies.

How often it fires. It is conditional: it collides whenever the existing index sits on a node that is still among the emptiest. Two runs with identical setup, differing only in which node holds the existing index:

A  existing index on node 1780 (19.0 GiB, still emptiest)
   -> new replica plans [1780]  -> COLLISION = true

B  existing index on node 1779 (4.0 GiB, no longer emptiest)
   -> new replica plans [1778]  -> COLLISION = false

The condition is common rather than rare. The existing replica was itself placed by the same emptiest-first rule, so it started on a top node, and its own reservation pushes that node down the ordering only by its own size. A small namespace reserving a few GB on a large node stays near the top. Large namespaces are better protected because their reservation is bigger.

Namespaces at 0 replicas are unaffected. When both replicas are planned in the same run, replica_plans accumulates between them and the nodes come out disjoint. Verified end to end — a fresh namespace with a desired count of 2 produced two replicas on distinct nodes with no errors.

How it presents. The plan is returned clean, with an empty errors array, and lands in the create bucket. ProvisioningService's Index.insert_all! then raises PG::UniqueViolation, which process_replica rescues as a generic StandardError and aggregates into the plan errors. The surrounding transaction rolls back, so no replica and no indices are created, and update_enabled_namespace stamps last_rollout_failed_at, which blocks retries through EnabledNamespace.with_rollout_allowed until the retry interval elapses. Externally the second replica silently never appears.

What is the expected correct behavior?

PlanningService.plan should never return a plan placing an index on a node that already holds an index for the same enabled namespace, whether that index was planned in the current run or already exists in the database. Raising zoekt_default_number_of_replicas should provision second replicas onto nodes disjoint from the existing ones, or record an actionable error when no such node has capacity.

Relevant logs and/or screenshots

Planning a second replica for a namespace that already has one provisioned:

existing replicas in DB: 1, on nodes [1767]
action=create replicas_planned=1
new replica would use nodes: [1767]
COLLIDES WITH EXISTING: true
errors=[]

The empty errors array is the important part — the plan reports success and the failure only surfaces at provisioning.

Possible fixes

  1. Seed the exclusion set from the database as well as the current run, for example by adding enabled_namespace.indices.pluck(:zoekt_node_id) to the node ids already excluded. Small change, and it makes the planner consistent with the constraint.
  2. Treat a namespace whose desired replica count has increased as a full re-plan rather than a delta, so all replicas are planned together and the existing ones are visible. Larger behavioural change.
  3. Add a guard in ProvisioningService filtering out any planned index whose node already holds one for the namespace, so an inconsistent plan degrades rather than raising.

A related issue is already open for a different trigger in the same area, where the planner reuses a node within a single replica. That one is reachable today; this one requires the replica count to be raised first.