## Problem to solve A `Search::Zoekt::Repository` can be `ready` in the database with no shard file on any node. Nothing detects that state, so exact code search returns nothing for the project while every operator surface reports healthy. The reporter on [issue #587507](https://gitlab.com/gitlab-org/gitlab/-/issues/587507) measured 176 non-empty projects in that state on a self-managed instance, all of them transferred between groups, none visible to the admin surfaces. The indexer-side race is fixed by [gitlab-org/gitlab-zoekt-indexer!1065](https://gitlab.com/gitlab-org/gitlab-zoekt-indexer/-/merge_requests/1065), which keeps a shard written after its delete marker. That stops new divergence. It repairs nothing already diverged, and it does not make an existing divergence visible. Two reasons nothing catches this today, both at `6dfe021908c05770e66a341221738e00b3888030`: - `ready` is written from the indexer callback and never re-checked against disk (`ee/app/services/search/zoekt/callback_service.rb:50`). A shard removed after that callback leaves the row untouched. - The only repository-level reconciliation is `should_be_marked_as_orphaned`, which tests `project_id IS NULL` (`ee/app/models/search/zoekt/repository.rb:46`). The `SchedulingService` config has no check that reads node state (`ee/app/services/search/zoekt/scheduling_service.rb:11-121`). `zoekt_repositories.index_file_count` (`db/structure.sql:36000`), populated per callback (`ee/app/services/search/zoekt/callback_service.rb:52-54`), is a node-reported shard count and looks like a free database-only signal. It is not one: it records what existed at the last successful index, which is exactly the moment before the shard is removed. ## Proposal **A cron reconciliation, not a rake task.** The loss on [issue #587507](https://gitlab.com/gitlab-org/gitlab/-/issues/587507) went unnoticed for months. An on-demand task only finds the divergence once somebody already suspects it, which is the condition that did not hold. Add the check to `SchedulingService::CONFIG` with a long `period` and one node per tick, following `repo_to_reindex_check` (`ee/app/services/search/zoekt/scheduling_service.rb:78-99`), rather than a new top-level cron worker. **Report, do not repair.** Detection emits a count and the offending project identifiers to a gauge, the structured log, and `gitlab-rake gitlab:zoekt:health`. Repair stays an explicit operator action, shaped like the existing `gitlab:zoekt:reindex_failed_projects` (`ee/lib/tasks/gitlab/zoekt.rake:70-74`), calling `Search::Zoekt::IndexingTaskService.execute(project_id, :force_index_repo)`. Automatic repair is the wrong default because a false positive costs a full re-index of the project. A listing taken while an index is in flight, a node restarted mid-sweep, or a partial response all produce false positives, and a systematic one on a large index would enqueue thousands of `force_index_repo` tasks against the node that is already unhealthy. The failure mode of reporting is a stale number in a log; the failure mode of repairing is a self-inflicted load event. **Two phases, because the bulk listing is not trustworthy enough to act on alone.** 1. Bulk candidate listing per node. `GET /debug/ls` exists (`internal/server/routes.go:34` at `9a5f4c555f8208cef067b109f90c0023b816e9aa`) and returns every file under the index directory with path, size and mtime (`internal/debug_ls/debug_ls.go:16-41`). Shard names are prefixed with the project id (`internal/zoekt/zoekt.go:56-66`), so the basenames yield the set present on disk. Diffing that against `Search::Zoekt::Repository.ready` for the indices on that node gives candidates. 2. Confirm each candidate individually with `GET /index/{id}`, which answers 404 when no shard metadata exists for that project (`internal/server/server.go:115-123`). Only a candidate that fails both checks is reported. **On the cost of a periodic sweep.** The directory walk itself is not new load: the node already runs `filepath.WalkDir` over the whole index directory every ten minutes to refresh the size cache it reports as `disk.indexed` (`internal/task_request/task_request.go:31,102`, `internal/disk_stats/disk_stats.go:79-96`). A daily reconciliation adds a small fraction to a walk the node performs 144 times a day. What is not free is the response. `/debug/ls` is unfiltered and unpaginated: it returns one JSON object per file, roughly 135 bytes each, built fully in memory before serialising. A large index returns megabytes per node per sweep, most of it fields the reconciliation does not read. Before this ships, the indexer should expose a listing that returns shard identifiers only; `/debug/ls` is a debug route and is the wrong contract to build a scheduled consumer on. ## Out of scope Repairing the projects already diverged. That is the operator rake task above, run once the detection has produced a list.