feat(orbit-cli): index with the incremental engine

Aaron Algutifanrequested to merge
aalgutifan/orbit-cli-incremental into main

What does this MR do and why?

orbit index now runs the incremental engine. The first run indexes everything and saves a snapshot per language family. Later runs re-index only the files that changed since then, uncommitted edits included, so re-indexing a large repository after a few edits takes seconds, not minutes. A missing snapshot or database falls back to a full index.

This replaces the old index engine and folds the separate reindex from !2664 (closed) into index.

Related Issues

Relates to !2664 (closed), which this carries forward onto current main.

Testing

New integration test incremental_index_follows_the_working_tree: an uncommitted edit, a rename, and a new language family each land in the graph on the next orbit index, and the file list matches a full index.

Checked by hand with the built binary on a repository with Rust and Ruby files: the first index, an edit, and a re-index.

Performance Analysis

  • This merge request does not introduce any performance regression. If a performance regression is expected, explain why.

A re-index after edits only parses the changed files and their importers. The first index of a repository costs the same as before.

Agent context: long-form analysis, file-by-file walkthroughs, profiler output, alternatives considered
  • commands/index/incremental/: per-family snapshots under var/<project>/, saved state in the manifest's _orbit_incremental_state, Project::index_all and Project::index_changed.
  • index takes the changed-files path when the repository has a saved state and the graph database exists, and --db is not set. It falls back to a full index if that path fails.
  • A manifest such as Cargo.toml is input to every family whose rules read it. The exporter keeps the first gl_file row and file containment edge, as it already did for directories, so it is written once.
  • The old engine's --threads flag is gone; --budget caps the seconds a language family may take.
  • The setup instructions no longer describe the graph from the old engine's node and edge kinds; that text had no placeholder left in the setup template.

A follow-up MR adds query-time refresh, so grep and context see edits without a re-index, along with the GitLab-backed grep and context.

Merge request reports