Stop the Zoekt rollout from looping on namespaces with orphaned projects

Ravi Kumarrequested to merge
zoekt-rollout-empty-replica-loop into master

What does this MR do and why?

Search::Zoekt::PlanningService#simulate_replica_plan skips project namespaces whose project is missing or has no statistics. When every project namespace in an enabled namespace is skipped this way, the method still planned a replica with zero indices, instead of falling back to create_empty_replica.

generate picked this path because fetch_project_namespaces.exists? uses with_project_statistics, which preloads statistics but doesn't filter on them, so it returns true even when no project namespace has a project with statistics.

The replica with no indices still got created by ProvisioningService. Index.insert_all!([]) is a no-op on an empty list, but the replica was still counted as a success and last_rollout_failed_at was cleared. On the next RolloutService run, cleanup_replicas_without_indices deleted the index-less replica, so the namespace was back to zero replicas and got selected again. With zoekt_rollout_batch_size at 1, this one namespace took the only rollout slot every run. On GitLab.com this stalled the Zoekt rollout, with about 116,915 namespaces waiting for their first index.

The fix: after the project namespace loop, if no indices were planned, fall back to create_empty_replica, the same path already used when there are no project namespaces at all. This plans one empty index that reaches ready, so the namespace is rolled out and stops blocking the queue.

zoekt_rollout_batch_size, the database health deferral of RolloutWorker, and why project namespaces exist without a project are out of scope and tracked in the linked issue.

References

Related to #630717

Screenshots or screen recordings

Backend only, no UI change.

How to set up and validate locally

  1. In a GDK with Zoekt set up, create a group with a project, and enable Zoekt for the group (create a Search::Zoekt::EnabledNamespace).
  2. In the Rails console, make the group's only project unindexable by deleting its statistics row: project.statistics.delete.
  3. Run: 
    Search::Zoekt::PlanningService.plan(
  enabled_namespaces: [enabled_namespace],
  nodes: Search::Zoekt::Node.available_for_search_indexing,
  max_indices_per_replica: Search::Zoekt::MAX_INDICES_PER_REPLICA
)
    Check that the planned replica has one index with a nil..nil projects range. Before the fix, the planned replica had an empty indices list.
  4. Optionally, with zoekt_indexing_paused set to false, run Search::Zoekt::RolloutWorker.new.perform, then let the Zoekt scheduling worker run initial indexing and "mark ready", and confirm the namespace has a replica whose index becomes ready.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

🤖 Generated with Claude Code

Merge request reports