Loading
Stop code repositories getting stuck in embedding indexing
Description
Some semantic code search repositories stay in embedding_indexing_in_progress forever. They never become ready, and they get no incremental indexing, so later pushes are not indexed.
This MR fixes the two causes:
- The initial run queues no chunks. The indexer succeeds, but the repository has nothing to index, for example only binary or large files. No queued item is set, and
MarkRepositoryAsReadyEventWorkerskips repositories without one. Now a run that queues no chunks moves the repository toready, as an empty repository does. - A deleted repository is indexed again.
RepositoryDeleteWorkerremoves the documents but keepslast_commit.AdHocIndexingWorkerthen starts the next run from that commit, so the indexer indexes only the changes since then. With no new commits, nothing is indexed. With new commits, only the changed files are indexed. Nowlast_commitis reset when the repository leaves thedeletedstate, so the next run indexes the full repository.
The repositories that are stuck today need a separate cleanup after this is deployed, tracked in https://gitlab.com/gitlab-org/gitlab/-/work_items/629396.
Related to #630967 (closed).
Edited by Arturo Herrero