Stop code repositories getting stuck in embedding indexing

Description

Some semantic code search repositories stay in embedding_indexing_in_progress forever. They never become ready, and they get no incremental indexing, so later pushes are not indexed.

This MR fixes the two causes:

  • The initial run queues no chunks. The indexer succeeds, but the repository has nothing to index, for example only binary or large files. No queued item is set, and MarkRepositoryAsReadyEventWorker skips repositories without one. Now a run that queues no chunks moves the repository to ready, as an empty repository does.
  • A deleted repository is indexed again. RepositoryDeleteWorker removes the documents but keeps last_commit. AdHocIndexingWorker then starts the next run from that commit, so the indexer indexes only the changes since then. With no new commits, nothing is indexed. With new commits, only the changed files are indexed. Now last_commit is reset when the repository leaves the deleted state, so the next run indexes the full repository.

The repositories that are stuck today need a separate cleanup after this is deployed, tracked in https://gitlab.com/gitlab-org/gitlab/-/work_items/629396.

Related to #630967 (closed).

Edited by Arturo Herrero

Merge request reports

Loading
Loading