## Summary Zoekt rollout has stalled since 09:40:58 UTC on 2026-09-24. One namespace is stuck in a loop, repeatedly creating replicas with no indices. With batch size 1, it takes the only rollout slot every run, blocking about 116,915 namespaces waiting for their first index. The oldest has waited since 2026-09-07. ## Root cause 1. The looping namespace's only project namespace has no project (`project_namespace.project` is nil). 2. `simulate_replica_plan` runs `next unless stats` when there's no project — it skips the index but still plans a replica, so the replica has zero indices. 3. `PlanningService#generate` picked `simulate_replica_plan` over `create_empty_replica` because `fetch_project_namespaces.exists?` was true — but `with_project_statistics` is a preload, not a filter, so it matches project namespaces with no project too. 4. `process_indices!` calls `Index.insert_all!([])`; in Rails 7.2 this returns early without raising, so the replica commits as a success and `last_rollout_failed_at` is cleared, not set. 5. Every run, `cleanup_replicas_without_indices` deletes this replica before selection, so with batch size 1 it takes the only slot again. 6. It gets an empty replica again, `re_enqueue` fires, and the cycle repeats, blocking everything behind it. ## Contributing factors - Un-indexable since 2026-09-07, yet the rollout moved past it until this morning. Likely, not proven: `with_mismatched_replicas` has no ORDER BY within a 5,000-id batch, so while many namespaces in the batch were mismatched others often came first; as the batch emptied, this namespace (reset by cleanup right before selection) started winning every run. - `zoekt_rollout_batch_size` is 1 (default 32), so one bad namespace takes 100% of a run's capacity. - Old namespaces rebuilding after index eviction have low ids and are scanned first, so they keep getting processed, which masks the stall. - `with_missing_indices` can't tell "waiting" from "looping", so it hid the stall as a normal backlog. ## What was ruled out | Hypothesis | Finding | |---|---| | Rollout blocked flag | Ruled out — none flagged. | | Prior rollout failure | Ruled out — `last_rollout_failed_at` unset. | | Sidekiq queue backlog | Ruled out — size and latency both 0. | | Indexing paused | Ruled out. | | Search nodes offline | Ruled out — 38/38 online. | ## Evidence - Last backlog index created 09:40:58 UTC 2026-09-24; no backlog progress since. - Empty replicas for the looping namespace at 12:12:12 and 15:05:21 UTC; `last_rollout_failed_at` stayed nil. - Replica ids advanced ~117 between 13:30 and 15:05 UTC while only one real index appeared — the loop likely runs often (indicative, not proof). - 61 replicas in the last 24 hours: backlog progress before the stall plus eviction rebuilds (the loop's own replicas are deleted by cleanup, so they don't show here). - The next namespace in line has zero replicas, a project with statistics and a clean plan, but was never rolled out. - `RolloutWorker`'s cron fires on schedule; its scheduled job was `deferred_by: "database_health_check"` (`deferred_count: 1`). ## Unblock Both are **production writes** needing a decision first. Either unblocks the rollout until the code fix ships. 1. Remove or disable the looping namespace's enabled-namespace record (note: `SaasRolloutEventWorker` re-creates records for eligible namespaces every 2 hours, so this may only be temporary). 2. Fix or clean up its orphaned project namespace. ## Proposal 1. In `PlanningService`, choose `simulate_replica_plan` vs. `create_empty_replica` by whether a project namespace has a real project with statistics — not just whether one exists. 2. In `process_indices!`, treat an empty indices plan as a failure: set `last_rollout_failed_at` and wait out the retry interval. 3. Raise `zoekt_rollout_batch_size` above 1 in steps, after confirming why it was lowered. 4. Once the deferral signal is known, narrow `RolloutWorker`'s watched tables. 5. Optional: query no-replica namespaces directly instead of scanning from the lowest id. Out of scope: why project namespaces have no project. Track rollout-per-day vs. enabled-per-day to catch stalls sooner. ## Open questions - Which health signal is deferring `RolloutWorker` jobs? - Why is `zoekt_rollout_batch_size` set to 1? - How many namespaces are stuck in this loop right now (`Search::Zoekt::Replica.without_indices`)? ## References - `ee/app/services/search/zoekt/rollout_service.rb` — cleanup before selection. - `ee/app/services/search/zoekt/selection_service.rb` — scans from the lowest id, stops at the batch size. - `ee/app/models/search/zoekt/enabled_namespace.rb` — `each_batch_with_mismatched_replicas` / `with_mismatched_replicas` (no ORDER BY within a batch), `with_missing_indices`. - `ee/app/services/search/zoekt/planning_service.rb` — wrong plan branch. - `ee/app/services/search/zoekt/provisioning_service.rb` — empty insert silently succeeds. - `app/models/namespace.rb` — `with_project_statistics`, the preload-not-filter. - `ee/app/workers/search/zoekt/rollout_worker.rb` — cron worker. - `ee/app/workers/search/zoekt/saas_rollout_event_worker.rb` — enables namespaces. - `lib/gitlab/sidekiq_middleware/skip_jobs.rb` — health deferral. <details><summary>Console checks used</summary> ```ruby missing = Search::Zoekt::EnabledNamespace.with_missing_indices missing.count; missing.with_rollout_blocked.count; missing.where.not(last_rollout_failed_at: nil).count Search::Zoekt::EnabledNamespace.where(created_at: 1.day.ago..).count Search::Zoekt::Replica.where(created_at: 1.day.ago..).count Search::Zoekt::Index.where('zoekt_enabled_namespace_id > ?', <looping enabled namespace id>).maximum(:created_at) ns = Search::Zoekt::EnabledNamespace.find(<looping enabled namespace id>) ns.replicas.map { |r| [r.id, r.state, r.created_at, r.indices.count] } Namespace.by_root_id(ns.root_namespace_id).project_namespaces.map { |pn| [pn.id, pn.project&.id, pn.project&.statistics.present?] } Search::Zoekt::PlanningService.plan(enabled_namespaces: [ns], nodes: Search::Zoekt::Node.available_for_search_indexing, max_indices_per_replica: Search::Zoekt::MAX_INDICES_PER_REPLICA) Search::Zoekt::SelectionService.execute(max_batch_size: 1).enabled_namespaces.map(&:id) Gitlab::CurrentSettings.zoekt_rollout_batch_size Sidekiq::Cron::Job.find('search_zoekt_rollout_worker') Gitlab::SidekiqSharding::Router.route(Search::Zoekt::RolloutWorker) { Sidekiq::ScheduledSet.new.scan('Search::Zoekt::RolloutWorker').map { |j| [j.at, j.item['deferred_by'], j.item['deferred_count']] } } ``` </details>