🤖 AI-authored change. ## Problem to solve Zoekt index and task management sits in the Rails monolith, so every change pays the monolith's review and CI cost. Extract it into a drop-in replacement from the indexer nodes' point of view. The node-facing contract is two endpoints in a 117-line file, and nodes pull, so Rails never pushes. Behind it sit 125 Zoekt files, 9,337 lines: 55 move, 31 mirror because Rails still reads them, 39 stay. Citations: [evidence pack](https://gitlab.com/gitlab-org/gitlab/-/work_items/628774#note_3822674737) Four hazards shape the phase order. A `truncate: true` heartbeat reply erases a node's index directory: 38 nodes, 3Ti each, ~48 hours to reindex. Rails reaps a node after 10 minutes without a heartbeat, not the code's 12-hour default, and reaping chains into that wipe: rolling back later than 10 minutes is itself the trigger. Split-brain is silent: a foreign callback returns with no log or metric, and a colliding task id marks the wrong repository indexed. Three tables are read on the search hot path, so they mirror rather than move. ## Proposal Decided: - The code lands in [zoekt-orchestrator](https://gitlab.com/gitlab-org/search-team/zoekt-orchestrator). - Placement stays in Rails through phase 1, which needs sizes only Rails holds, and Rails keeps a fallback planner permanently. - Phase 3 ships on HTTP, so the absence of NATS in gprd cannot block the sequence. - "Per-namespace" is two switches: a chart value moves traffic per StatefulSet, an ops flag fences Rails writes. Open: - Request a `gitlab-org/security/` mirror? Zoekt has no private security-fix path today. - Is the internal gateway path-routable, or opaque L4? A path route makes cutover atomic in under 5s, but the gateway also fronts KAS and Mailroom, so the call is Infrastructure's. Each phase lands a merged spec in [artifacts](https://gitlab.com/gitlab-org/search-team/artifacts) before its implementation MR. Sequencing is in `is_blocked_by` links, not dates. Placement and the transport swap are both held by the cutover, and neither holds the other. - [Phase 0 — disable lost-node threshold](https://gitlab.com/gitlab-org/gitlab/-/work_items/628774) - [Phase 1 — shadow orchestrator, seeded uuids](https://gitlab.com/gitlab-org/gitlab/-/work_items/628775) - [Phase 2 — own Postgres, regenerated tasks](https://gitlab.com/gitlab-org/gitlab/-/work_items/628776) - [Phase 3 — publish the eight events](https://gitlab.com/gitlab-org/gitlab/-/work_items/628777) - [Phase 4 — fence Rails writes, one ns](https://gitlab.com/gitlab-org/gitlab/-/work_items/628778) - [Phase 5 — cut node traffic over](https://gitlab.com/gitlab-org/gitlab/-/work_items/628779) - [Phase 6 — move placement out of Rails](https://gitlab.com/gitlab-org/gitlab/-/work_items/628780) - [Phase 7 — JetStream, restore threshold](https://gitlab.com/gitlab-org/gitlab/-/work_items/628781) 🤖 *Automated change. Mention `@johnmason` to leave feedback.* <details> <summary>🤖 Mechanism and citations</summary> ## Mechanism All citations are against `gitlab-org/gitlab` at `02c02def9c94a71fc1a0234e395c290354660966`. No Zoekt file differs between that commit and `origin/master` (`56cb98bc`), so the line numbers do not drift. The two counts in the body measure different things: 117 lines is the node-facing API file, 9,337 lines is the Zoekt implementation behind it. ## Citations - Node contract: `ee/lib/api/internal/search/zoekt.rb`, callback and heartbeat. Measured 25.0 req/s, p95 0.569s, gprd Mimir. - Index wipe: `zoekt.rb:97,101` -> `task_request_response.go:32-56` -> `file_cleaner.go:141-163`. - Lost-node fuse: `settings.rb:20` holds the 12-hour default; `gitlab-com/runbooks` `docs/zoekt/README.md:294-300` records the 10-minute production value and the chain to the wipe. Row deletion: `lost_node_event_worker.rb:16-41`. - Silent split-brain: `callback_service.rb:20,29-34` and `:43-69`. - Hot path: `zoekt_searchable.rb:46-49`, `node_selector.rb:48-64`, `replica_selector.rb:23-46`. ## What I did not verify - No gprd go-live date for NATS appears in anything fetched. - No gstg metric series exist for Zoekt, so staging figures are unverified. - Phase 5's task-throughput tolerance still needs a number. - Whether the internal gateway supports per-path routing is unanswered. - Gitaly payload data is not computable by the orchestrator and no design is settled. </details>