Stop the Zoekt planner reusing a node within a replica (!253655) · Merge requests · GitLab.org / GitLab

What does this MR do and why?

zoekt_indices has a unique index u_zoekt_indices_zoekt_enabled_namespace_id_and_zoekt_node_id on (zoekt_enabled_namespace_id, zoekt_node_id): two indices of the same namespace cannot share a node, and so neither can two indices of one replica. Search::Zoekt::PlanningService could still produce a plan that violates this, and provisioning it raises PG::UniqueViolation from ProvisioningService's Index.insert_all!.

Root cause, in ee/app/services/search/zoekt/planning_service.rb: assign_project_to_index only retires a node into @exhausted_node_ids when last_index is present:

@exhausted_node_ids.add(node[:id]) if last_index # If this is the first project of the replica then skip

For the first index of a replica, replica_indices is empty, so last_index is nil and the node is never retired there. It stays eligible for the whole packing loop and is only retired at the end by add_replica_plan. find_best_node is a first-match scan over nodes, sorted by unclaimed space descending in format_nodes and never re-sorted, so the emptiest node stays first in the array and can be selected again once the current index fills up.

The fix

@exhausted_node_ids was doing two jobs at once: excluding nodes used by earlier replicas, and gating whether packing could continue at all. It was also deliberately incomplete — the first index of a replica never retired its node — and that hole is precisely what let a node take a second index.

It is replaced by used_node_ids, derived rather than accumulated:

# Every node this namespace already has an index on: the ones already persisted, plus all
# replicas planned so far and the one being packed. zoekt_indices is unique on
# (zoekt_enabled_namespace_id, zoekt_node_id), so none of them can take another index.
def used_node_ids(replica_indices)
  planned = replica_plans.flat_map { |rp| rp[:indices].pluck(:node_id) }
  persisted_node_ids + planned + replica_indices.pluck(:node_id)
end

def persisted_node_ids
  enabled_namespace.nodes.pluck_primary_key
end
strong_memoize_attr :persisted_node_ids

replica_plans already records completed replicas and format_index keeps :node_id, so no state is lost there. persisted_node_ids adds the nodes the namespace already has indices on, queried once per namespace and memoized. used_node_ids is consulted only where a new index is opened. The added query runs only on the :create path; generate already runs a per-namespace enabled_namespace.replicas.size query, and RolloutService caps the batch at 128 namespaces (DEFAULT_OPTIONS[:batch_size]), so the added cost is bounded.

The gate disappears with it. assign_to_node previously called find_best_node for every project and errored when it returned nil, even for projects that would only extend the current index. Now a node is looked up when one is actually needed, so assign_to_node collapses into assign_project_to_index, which also loses its unused node parameter and the if last_index special case.

That has a second, deliberate effect: a project that fits the current index is now packed even when no other node could open a new one. Previously that failed with node_unavailable. Measured on a two-node scenario where the third project fits index 2 but no node can host a third index:

before: create=0 failures=1  packed=13,392,412,672  node_unavailable
after : create=1 failures=0  packed=14,466,154,496  errors=[]

Note what that meant before: accumulate_error puts the namespace's plan in failures, and ProvisioningService only processes create, so a single dropped project meant the namespace provisioned nothing. Those namespaces now provision, so more of them claim storage. Cross-replica separation is unchanged and covered by existing specs.

node_unavailable_details now distinguishes the two causes instead of sharing one message:

def node_unavailable_details(stats, excluded_node_ids)
  if excluded_node_ids.any? && find_best_node(stats, []).present?
    "Every node with capacity for project #{stats.project_id} with size #{scaled_size(stats)} " \
      "already holds an index for this namespace"
  else
    "No node can accommodate project #{stats.project_id} with size #{scaled_size(stats)}"
  end
end

The extra unfiltered scan runs only on the error path, and the error type stays :node_unavailable. Previously a single message could not tell "nothing has room" apart from "something has room but already holds an index for this namespace", which matters because this class of failure was hard to diagnose from production data. This is also why find_best_node's second parameter is now excluded_node_ids rather than used_node_ids — it no longer shadows the used_node_ids method above.

Why not retire every node instead

Deleting the if last_index guard on its own looks like the smaller fix, but @exhausted_node_ids also gated whether packing could continue at all. Retiring the first index's node immediately breaks single-node instances. Measured with three 1 GiB projects on one 10 GiB node:

guard removed : create=0 failures=1   1 index, 1.07 GB packed, 2 x node_unavailable
this MR       : create=1 failures=0   1 index, 3.22 GB packed, no errors

Nor is the node's free space forfeited. max_storage_bytes is the node's unclaimed space at index creation, so the owning index grows into all of it — nine 1 GiB projects pack into a single index on a 10 GiB node. And because the exclusion is per namespace, another namespace can still use the remainder: two namespaces sharing one 10 GiB node claimed 6 GiB and 3 GiB respectively in the same plan.

What the guard could not legitimately buy is a second index on that node, since the unique index is on (zoekt_enabled_namespace_id, zoekt_node_id) and omits zoekt_replica_id.

Observed on production

A dry-run plan for namespace 8338830 produced 10 indices across only 9 distinct nodes, with node 1000198 holding both index 1 (range (..49853664)) and index 4 (range (79422119..80421166)).

The arithmetic confirms the mechanism: node 1000198 had 71,859,053,179 bytes unclaimed, index 1 took 71,244,546,477, and the remainder — 614,506,702 — is exactly index 4's max_storage_bytes. simulate_index sizes max_storage_bytes from the node's current unclaimed space, so index 4 was sized against 1000198's leftover and no other node's.

That specific plan failed earlier with index_limit_exceeded and never reached provisioning, but that failure is incidental to the bug.

Verified on production

The same namespace 8338830 re-planned with this change applied. Ten indices across ten distinct nodes — previously ten indices across nine, with node 1000198 appearing twice:

	before	after
indices	10	10
distinct nodes	9	10
duplicate node	`1000198` twice	none
node capacity offered	162.3 GB	140.4 GB
assigned	154.7 GB	128.6 GB

The namespace still terminates on index_limit_exceeded, and coverage fell — but for reasons unrelated to this change. Between the two runs the fleet tightened: node 1000198 went from 71.9 GB unclaimed to 16.2 GB as other namespaces claimed it, and the largest node available dropped from 71.9 GB to 26.7 GB. With no large node left, each of the ten indices holds less.

Worth noting the plan left 11.8 GB unused across those ten nodes. It did not run out of space, it ran out of index slots — MAX_INDICES_PER_REPLICA is 10. That is a separate constraint from the one this MR fixes.

Full plan output (nodes array omitted for length)

{:total_required_storage_bytes=>0,
 :create=>[],
 :destroy=>[],
 :unchanged=>[],
 :failures=>
  [{:namespace_id=>8338830,
    :enabled_namespace_id=>2159361,
    :namespace_required_storage_bytes=>128611604172.37766,
    :replicas=>
     [{:indices=>
        [{:node_id=>1000213,
          :node_name=>"gitlab-gitlab-zoekt-16",
          :projects=>{:project_namespace_id_from=>nil, :project_namespace_id_to=>45983412},
          :required_storage_bytes=>26221480096.292088,
          :max_storage_bytes=>0.26693823673e11},
         {:node_id=>2000230,
          :node_name=>"gitlab-gitlab-zoekt-33",
          :projects=>{:project_namespace_id_from=>45983413, :project_namespace_id_to=>47810928},
          :required_storage_bytes=>22837749224.6378,
          :max_storage_bytes=>0.23172825725e11},
         {:node_id=>1000212,
          :node_name=>"gitlab-gitlab-zoekt-17",
          :projects=>{:project_namespace_id_from=>47810929, :project_namespace_id_to=>49853664},
          :required_storage_bytes=>22201638107.15658,
          :max_storage_bytes=>0.2245918377e11},
         {:node_id=>1000198,
          :node_name=>"gitlab-gitlab-zoekt-31",
          :projects=>{:project_namespace_id_from=>49853665, :project_namespace_id_to=>55280238},
          :required_storage_bytes=>13477472494.025707,
          :max_storage_bytes=>0.16203699486e11},
         {:node_id=>1000227,
          :node_name=>"gitlab-gitlab-zoekt-2",
          :projects=>{:project_namespace_id_from=>55280239, :project_namespace_id_to=>58158378},
          :required_storage_bytes=>13952807884.194174,
          :max_storage_bytes=>0.14076355401e11},
         {:node_id=>1000199,
          :node_name=>"gitlab-gitlab-zoekt-30",
          :projects=>{:project_namespace_id_from=>58158379, :project_namespace_id_to=>68262942},
          :required_storage_bytes=>12688431923.180077,
          :max_storage_bytes=>0.12698758869e11},
         {:node_id=>1000195,
          :node_name=>"gitlab-gitlab-zoekt-34",
          :projects=>{:project_namespace_id_from=>68262943, :project_namespace_id_to=>76206658},
          :required_storage_bytes=>6994489728.867184,
          :max_storage_bytes=>0.7240854371e10},
         {:node_id=>1000200,
          :node_name=>"gitlab-gitlab-zoekt-29",
          :projects=>{:project_namespace_id_from=>76206659, :project_namespace_id_to=>78828512},
          :required_storage_bytes=>3932840323.054954,
          :max_storage_bytes=>0.6510313892e10},
         {:node_id=>1000228,
          :node_name=>"gitlab-gitlab-zoekt-1",
          :projects=>{:project_namespace_id_from=>78828513, :project_namespace_id_to=>79109945},
          :required_storage_bytes=>5155850527.954177,
          :max_storage_bytes=>0.5826294185e10},
         {:node_id=>1000204,
          :node_name=>"gitlab-gitlab-zoekt-25",
          :projects=>{:project_namespace_id_from=>79109946, :project_namespace_id_to=>nil},
          :required_storage_bytes=>1148843863.0149124,
          :max_storage_bytes=>0.5543406539e10}]}],
    :errors=>[{:namespace_id=>8338830, :replica_idx=>0, :type=>:index_limit_exceeded, :details=>"Replica reached maximum index limit (10)"}],
    :action=>:create}]}

The duplicate is not limited to failing plans

Running the unfixed planner on a three-node scenario produced a plan in the create bucket with an empty errors array that still contained a repeated node id, and provisioning it failed for real:

plan buckets: create=1 failures=0 unchanged=0
errors: []
indices=3 node_ids=[244, 245, 244] distinct=2

error: PG::UniqueViolation: ERROR: duplicate key value violates unique constraint
       "u_zoekt_indices_zoekt_enabled_namespace_id_and_zoekt_node_id"
replicas now: 0
indices now : 0
enabled_namespace.last_rollout_failed_at: 2026-09-04 10:55:38 UTC

The duplicate needs at least three indices and cannot occur with fewer, which is why most rollouts are unaffected. Index 1 fills the first node and leaves a remainder; the project that overflowed it is by definition larger than that remainder, so index 2 always lands elsewhere. Index 3 is the first opportunity, because the project overflowing index 2 must exceed the second node's remainder yet may still fit the first node's. Namespaces that fit in one or two indices are immune.

When it does fire it fails quietly. ProvisioningService#process_replica rescues StandardError, so the violation is aggregated into the plan errors rather than raised, the transaction rolls back leaving no replica and no indices, and last_rollout_failed_at is stamped, which blocks retries via EnabledNamespace.with_rollout_allowed until the retry interval elapses. Externally it looks like a large namespace that will not provision.

Existing indices, and one gap left open

The planner now excludes nodes that already carry a persisted index for the namespace, so raising a namespace's replica count from 1 to 2 can no longer plan the new replica onto a node the existing one occupies.

One gap remains, out of scope for this MR: ProvisioningService#process_indices! still ends in a bare Index.insert_all! with no conflict handling, so the write layer has no defence in depth if a future planner change reintroduces a duplicate.

Test

Four new examples:

node reuse within a replica — three nodes of 10, 5 and 4 GiB and three projects sized so the first index leaves a remainder the third project fits into. Asserts three indices and no repeated node_id.
when every node with capacity is already in the replica — same setup without the third node, so the only node with room is already used. Asserts node_unavailable is recorded with the new "already holds an index for this namespace" message, and not the old "No node can accommodate" wording.
packing into an index whose node is the only one with capacity — covers the removed gate: asserts every project is packed and no failure is recorded.
nodes already holding a persisted index for the namespace — the namespace already has an index on node_a (10 GiB free, the emptiest node); node_b has 5 GiB; number_of_replicas_override is 2; one 1 GiB project. Asserts the added replica lands on node_b and no failure is recorded.

The first three were confirmed to fail against the unmodified planner and pass with the change. The fourth was verified differently, by dropping the persisted_node_ids + term from used_node_ids: with that term removed the planner picks node_a, the emptiest node, instead.

Each scopes its node list explicitly so the file's other node fixtures cannot supply an alternative and mask the behaviour.

Validation

ee/spec/services/search/zoekt/ — 433 examples, 0 failures
ee/spec/services/search/zoekt/planning_service_spec.rb — 35 examples, 0 failures
The three original examples were verified against the unmodified planner: 3 examples, 3 failures
The fourth example was verified by dropping the persisted_node_ids + term: it fails, with the planner picking node_a, the emptiest node
RuboCop clean on both files

References

Closes #627730 (closed)
Closes #627737

Screenshots or screen recordings

Not applicable — backend-only change with no UI surface.

How to set up and validate locally

Run bundle exec rspec ee/spec/services/search/zoekt/planning_service_spec.rb — 35 examples, including the four new ones.
Optionally check out the original planning_service.rb from master and re-run the first three new examples; all three fail.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited Sep 10, 2026 by Ravi Kumar

Stop the Zoekt planner reusing a node within a replica