Add support for Incremental Indexing
As part of the issue #79 (closed) We are adding support full indexing for the repositories.
And as part of #105 (closed), we are converting the full indexing to be use HTTP Server (GKG new service architecture).
As part of this issue we need to look into incremental indexing of repositories. That happens when an existing graph index exists for the Repository on the Zoekt Indexer node, when Forced Reindexing is not requested.
We need to look into The Knowledge Graph repo and see how the Incremental Indexing is implemented, and add bindings for that. At the moment we have only bindings for Full Indexing.
See this related code:
- https://gitlab.com/gitlab-org/rust/knowledge-graph/-/tree/main/crates/indexer-c-bindings?ref_type=heads
- https://gitlab.com/gitlab-org/rust/knowledge-graph/-/tree/main/crates/indexer/src/deployed?ref_type=heads
Update Rails Task
As part of this issue we need to add two more values to the exposed knowledge graph indexing tasks on Rails side to include
- Force Indexing. By default it's
false. It's onlytruewhen Rails detects mismatch between the currently indexed schema, and the new schema of GKG indexing server. - File Limit. This is necessary for when we pull files from Gitaly. If the files are over this file limit, we don't pull the content.
Both of them are implemented on Zoekt side as well.