2025-08-08: GSTG: Zoekt creates tasks in a loop

GSTG: Zoekt creates tasks in a loop (Severity 4)

Problem: After upgrading Zoekt to v1.2.0, the system began creating an unexpected and excessive number of indexing tasks, with currently 13,211,903 tasks pending.

Impact: This problem has caused significant increases in memory usage, up to 20 GB, and has slowed down the execution of reindexing queries.

Causes: The root cause of the problem is that the new zero-downtime logic in Zoekt creates tasks repeatedly without proper checks, and the existing database index was not being utilized for reindexing queries.

Response strategy: We have prepared and approved a merge request to fix the Zoekt reindexing logic and prevent excessive task creation by separating normal and zero-downtime indexing and adding safeguards. We implemented a new covering index to optimize the query for common use cases, which has been approved and set to merge. We paused indexing on staging and plan to clean up duplicate tasks and resume indexing after the merge request is fully implemented.

---

This ticket was created to track INC-3164, by incident.io 🔥