feat(code-graph-incremental): snapshots and reindex

What does this MR do and why?

Eighth slice of code-graph-incremental, stacked on !2641 (merged).

This is the "incremental indexing" MR. A graph can now be saved to disk and loaded back, and a change set (files added, modified, removed) re-indexes only those files: the change-set is removed from the graph, and what depended on them is marked for re-resolution, and only they go through parse, rewrite and link again.

The suite runner now executes incremental steps, so the YAML suites check the graph after each change, including after a save and load.

Relates to #1284. Reference implementation: !2454 (closed).

Testing

cargo test -p code-graph-incremental: 48 tests. New in tests/snapshot.rs: a snapshot round-trips trees, edges and files; a reindex right after a load keeps the import resolved and displays the right names (the rules must compile against the restored interner); a removed file leaves the graph, its definitions and cross-file edges with it.

cargo nextest run -p integration-tests-codegraph: incremental_suites 49/49, now including python/incremental/{basic,touch}.yaml, both with snapshot: true on the first step; suites 247/247 unchanged.

Performance Analysis

No runtime change: nothing calls this crate yet. Snapshots are rkyv with zstd level 3; on the Sentry repository in the reference branch, save added about 0.6GB peak RSS to a 1.1GB run.

  • This merge request does not introduce any performance regression. If a performance regression is expected, explain why.
Agent context: long-form analysis, file-by-file walkthroughs, profiler output, alternatives considered

What lands

src/pipeline/state.rs          State::save / State::load; Interner, Lang, Tree, Resolver snapshots (rkyv)
src/pipeline/phases.rs         Remap (ReindexInput -> Workset), remap()
src/templates.rs               reindex(): Remap, Each(Parse+Rewrite+Canonicalize+Link), Insert, Resolve
tests/snapshot.rs
crates/integration-tests-codegraph/
  src/assertions.rs            steps: Vec<IncrementalStep> { name, snapshot, add, modify, remove, tests }
  src/incremental.rs           the step loop; check() exports into a fresh DuckDB per step
  src/runner.rs                the code-graph runner refuses suites with steps
  fixtures_incremental/python/incremental/{basic,touch}.yaml

Files are copied from !2454 (closed) at their final shape.

State::load compiles the language rules into the interner restored from the snapshot (Env::with_lang), not into a fresh one. With a fresh interner the ids the rules use and the ids in the graph disagree, and the first reindex after a load exports wrong file paths and fqns while edge counts stay identical. reindex_after_load_keeps_resolution_and_names and the snapshot: true suites cover this.

Exported ids do not depend on tree positions, so Remap renumbering trees never changes a row id; only the row's own content does.

Base branch is michaelusa/code-graph-incremental-suites (!2641 (merged)); retarget to main once that merges.

MR stack

MR Contents
!2620 (merged) runner, budgets, errors, canonical alphabet, artifact and graph types
!2627 (merged) Tree, Interner, tree-sitter languages, Env, inventory, Prepare + Each + Parse
!2629 (merged) DSL engine, Rewrite + Canonicalize, Env.rules, Python rules
!2631 (merged) SSA, linker, Link + Insert
!2633 (merged) resolver, project tree, Resolve
!2638 (merged) export engine, Display + Export + Emit, index template
!2641 (merged) YAML suites through integration-tests-codegraph with a second runner
8 (this) State save/load, Remap, reindex template, incremental suites
9 remaining language rule files and their suites
10 orbit index --engine incremental, full index through the existing DuckDB sink
11 snapshot next to the DuckDB file, changed paths from git, incremental CLI
12 default flip per language; old pipeline and its fixtures retire

%{all_commits}

Edited by Michael Usachenko

Merge request reports

Loading
Loading