Manage failures in code indexing pipeline
Description
We need to manage failures in the code indexing pipeline. There are different parts of the system where things can fail: Gitaly calls, chunking process, and indexing operations. We need to clarify our failure handling strategy:
Low-level approach: Track individual chunk IDs and retry specific failed chunks. This preserves successful work but requires complex state management and tracking of what succeeded vs failed.
High-level approach: Abort entire file/repository operations on any failure and retry the complete process. This is simpler to implement but potentially wasteful as successful chunks get reprocessed.
Key considerations:
- Different failure types (connection errors, processing failures, adapter errors) may need different handling strategies
- Initial indexing vs incremental updates may have different tolerance for waste
- Need to define clear contracts between Rails, chucker, and indexer for error communication
The current consensus is to start with the high-level "fail entire operation" approach for simplicity, then evolve to more granular handling once the happy path is established.
Related to gitlab-elasticsearch-indexer!650 (comment 2552852802):
We may want to think a bit on what to do when things fail in this entire process: we should return a failed status to rails and mark the
Ai::ActiveContext::Repositoryas failed but what if some of the batches succeeded and some failed? Should we have a SchedulingService task that deletes all the current indexed documents for the repository and start from scratch? It could be wasteful. Otherwise we need to track what is successful and what's not like the ActiveContext gem does.