One namespace stuck in a Zoekt rollout loop blocks progress for 116,915 waiting namespaces

Summary

Zoekt rollout has stalled since 09:40:58 UTC on 2026-09-24. One namespace is stuck in a loop, repeatedly creating replicas with no indices. With batch size 1, it takes the only rollout slot every run, blocking about 116,915 namespaces waiting for their first index. The oldest has waited since 2026-09-07.

Root cause

  1. The looping namespace's only project namespace has no project (project_namespace.project is nil).
  2. simulate_replica_plan runs next unless stats when there's no project — it skips the index but still plans a replica, so the replica has zero indices.
  3. PlanningService#generate picked simulate_replica_plan over create_empty_replica because fetch_project_namespaces.exists? was true — but with_project_statistics is a preload, not a filter, so it matches project namespaces with no project too.
  4. process_indices! calls Index.insert_all!([]); in Rails 7.2 this returns early without raising, so the replica commits as a success and last_rollout_failed_at is cleared, not set.
  5. Every run, cleanup_replicas_without_indices deletes this replica before selection, so with batch size 1 it takes the only slot again.
  6. It gets an empty replica again, re_enqueue fires, and the cycle repeats, blocking everything behind it.

Contributing factors

  • Un-indexable since 2026-09-07, yet the rollout moved past it until this morning. Likely, not proven: with_mismatched_replicas has no ORDER BY within a 5,000-id batch, so while many namespaces in the batch were mismatched others often came first; as the batch emptied, this namespace (reset by cleanup right before selection) started winning every run.
  • zoekt_rollout_batch_size is 1 (default 32), so one bad namespace takes 100% of a run's capacity.
  • Old namespaces rebuilding after index eviction have low ids and are scanned first, so they keep getting processed, which masks the stall.
  • with_missing_indices can't tell "waiting" from "looping", so it hid the stall as a normal backlog.

What was ruled out

Hypothesis Finding
Rollout blocked flag Ruled out — none flagged.
Prior rollout failure Ruled out — last_rollout_failed_at unset.
Sidekiq queue backlog Ruled out — size and latency both 0.
Indexing paused Ruled out.
Search nodes offline Ruled out — 38/38 online.

Evidence

  • Last backlog index created 09:40:58 UTC 2026-09-24; no backlog progress since.
  • Empty replicas for the looping namespace at 12:12:12 and 15:05:21 UTC; last_rollout_failed_at stayed nil.
  • Replica ids advanced ~117 between 13:30 and 15:05 UTC while only one real index appeared — the loop likely runs often (indicative, not proof).
  • 61 replicas in the last 24 hours: backlog progress before the stall plus eviction rebuilds (the loop's own replicas are deleted by cleanup, so they don't show here).
  • The next namespace in line has zero replicas, a project with statistics and a clean plan, but was never rolled out.
  • RolloutWorker's cron fires on schedule; its scheduled job was deferred_by: "database_health_check" (deferred_count: 1).

Unblock

Both are production writes needing a decision first. Either unblocks the rollout until the code fix ships.

  1. Remove or disable the looping namespace's enabled-namespace record (note: SaasRolloutEventWorker re-creates records for eligible namespaces every 2 hours, so this may only be temporary).
  2. Fix or clean up its orphaned project namespace.

Proposal

  1. In PlanningService, choose simulate_replica_plan vs. create_empty_replica by whether a project namespace has a real project with statistics — not just whether one exists.
  2. In process_indices!, treat an empty indices plan as a failure: set last_rollout_failed_at and wait out the retry interval.
  3. Raise zoekt_rollout_batch_size above 1 in steps, after confirming why it was lowered.
  4. Once the deferral signal is known, narrow RolloutWorker's watched tables.
  5. Optional: query no-replica namespaces directly instead of scanning from the lowest id.

Out of scope: why project namespaces have no project. Track rollout-per-day vs. enabled-per-day to catch stalls sooner.

Open questions

  • Which health signal is deferring RolloutWorker jobs?
  • Why is zoekt_rollout_batch_size set to 1?
  • How many namespaces are stuck in this loop right now (Search::Zoekt::Replica.without_indices)?

References

  • ee/app/services/search/zoekt/rollout_service.rb — cleanup before selection.
  • ee/app/services/search/zoekt/selection_service.rb — scans from the lowest id, stops at the batch size.
  • ee/app/models/search/zoekt/enabled_namespace.rb — each_batch_with_mismatched_replicas / with_mismatched_replicas (no ORDER BY within a batch), with_missing_indices.
  • ee/app/services/search/zoekt/planning_service.rb — wrong plan branch.
  • ee/app/services/search/zoekt/provisioning_service.rb — empty insert silently succeeds.
  • app/models/namespace.rb — with_project_statistics, the preload-not-filter.
  • ee/app/workers/search/zoekt/rollout_worker.rb — cron worker.
  • ee/app/workers/search/zoekt/saas_rollout_event_worker.rb — enables namespaces.
  • lib/gitlab/sidekiq_middleware/skip_jobs.rb — health deferral.
Console checks used 
missing = Search::Zoekt::EnabledNamespace.with_missing_indices
missing.count; missing.with_rollout_blocked.count; missing.where.not(last_rollout_failed_at: nil).count
Search::Zoekt::EnabledNamespace.where(created_at: 1.day.ago..).count
Search::Zoekt::Replica.where(created_at: 1.day.ago..).count
Search::Zoekt::Index.where('zoekt_enabled_namespace_id > ?', <looping enabled namespace id>).maximum(:created_at)
ns = Search::Zoekt::EnabledNamespace.find(<looping enabled namespace id>)
ns.replicas.map { |r| [r.id, r.state, r.created_at, r.indices.count] }
Namespace.by_root_id(ns.root_namespace_id).project_namespaces.map { |pn| [pn.id, pn.project&.id, pn.project&.statistics.present?] }
Search::Zoekt::PlanningService.plan(enabled_namespaces: [ns], nodes: Search::Zoekt::Node.available_for_search_indexing, max_indices_per_replica: Search::Zoekt::MAX_INDICES_PER_REPLICA)
Search::Zoekt::SelectionService.execute(max_batch_size: 1).enabled_namespaces.map(&:id)
Gitlab::CurrentSettings.zoekt_rollout_batch_size
Sidekiq::Cron::Job.find('search_zoekt_rollout_worker')
Gitlab::SidekiqSharding::Router.route(Search::Zoekt::RolloutWorker) { Sidekiq::ScheduledSet.new.scan('Search::Zoekt::RolloutWorker').map { |j| [j.at, j.item['deferred_by'], j.item['deferred_count']] } }
Edited by Ravi Kumar