feat(code-graph-incremental): snapshots and reindex
What does this MR do and why?
Eighth slice of code-graph-incremental, stacked on !2641 (merged).
This is the "incremental indexing" MR. A graph can now be saved to disk and loaded back, and a change set (files added, modified, removed) re-indexes only those files: the change-set is removed from the graph, and what depended on them is marked for re-resolution, and only they go through parse, rewrite and link again.
The suite runner now executes incremental steps, so the YAML suites check the graph after each change, including after a save and load.
Related Issues
Relates to #1284. Reference implementation: !2454 (closed).
Testing
cargo test -p code-graph-incremental: 48 tests. New in tests/snapshot.rs: a snapshot round-trips trees, edges and files; a reindex right after a load keeps the import resolved and displays the right names (the rules must compile against the restored interner); a removed file leaves the graph, its definitions and cross-file edges with it.
cargo nextest run -p integration-tests-codegraph: incremental_suites 49/49, now including python/incremental/{basic,touch}.yaml, both with snapshot: true on the first step; suites 247/247 unchanged.
Performance Analysis
No runtime change: nothing calls this crate yet. Snapshots are rkyv with zstd level 3; on the Sentry repository in the reference branch, save added about 0.6GB peak RSS to a 1.1GB run.
- This merge request does not introduce any performance regression. If a performance regression is expected, explain why.
Agent context: long-form analysis, file-by-file walkthroughs, profiler output, alternatives considered
What lands
src/pipeline/state.rs State::save / State::load; Interner, Lang, Tree, Resolver snapshots (rkyv)
src/pipeline/phases.rs Remap (ReindexInput -> Workset), remap()
src/templates.rs reindex(): Remap, Each(Parse+Rewrite+Canonicalize+Link), Insert, Resolve
tests/snapshot.rs
crates/integration-tests-codegraph/
src/assertions.rs steps: Vec<IncrementalStep> { name, snapshot, add, modify, remove, tests }
src/incremental.rs the step loop; check() exports into a fresh DuckDB per step
src/runner.rs the code-graph runner refuses suites with steps
fixtures_incremental/python/incremental/{basic,touch}.yamlFiles are copied from !2454 (closed) at their final shape.
State::load compiles the language rules into the interner restored from the snapshot (Env::with_lang), not into a fresh one. With a fresh interner the ids the rules use and the ids in the graph disagree, and the first reindex after a load exports wrong file paths and fqns while edge counts stay identical. reindex_after_load_keeps_resolution_and_names and the snapshot: true suites cover this.
Exported ids do not depend on tree positions, so Remap renumbering trees never changes a row id; only the row's own content does.
Base branch is michaelusa/code-graph-incremental-suites (!2641 (merged)); retarget to main once that merges.
MR stack
| MR | Contents |
|---|---|
| !2620 (merged) | runner, budgets, errors, canonical alphabet, artifact and graph types |
| !2627 (merged) | Tree, Interner, tree-sitter languages, Env, inventory, Prepare + Each + Parse |
| !2629 (merged) | DSL engine, Rewrite + Canonicalize, Env.rules, Python rules |
| !2631 (merged) | SSA, linker, Link + Insert |
| !2633 (merged) | resolver, project tree, Resolve |
| !2638 (merged) | export engine, Display + Export + Emit, index template |
| !2641 (merged) | YAML suites through integration-tests-codegraph with a second runner |
| 8 (this) | State save/load, Remap, reindex template, incremental suites |
| 9 | remaining language rule files and their suites |
| 10 | orbit index --engine incremental, full index through the existing DuckDB sink |
| 11 | snapshot next to the DuckDB file, changed paths from git, incremental CLI |
| 12 | default flip per language; old pipeline and its fixtures retire |
%{all_commits}