feat(code-graph-incremental): export the graph as ontology tables

What does this MR do and why?

An incremental code indexing run now ends with a set of RecordBatch that correspond to the ontology's local tables: gl_file, gl_directory, gl_definition, gl_imported_symbol, gl_edge, with the same columns and the same content-hashed ids as production writes. Nothing is wired to the CLI or server yet.

Relates to #1284. Reference implementation: !2454 (closed).

Testing

cargo test -p code-graph-incremental: 45 tests. New in tests/export.rs: every file becomes a row with its size and reason, and src becomes a directory row; definitions, imports and edges export with the expected fqn, type, file and line; two runs over the same repository export identical ids in every table; Emit hands every table to the sink and keeps the graph.

cargo clippy -p code-graph-incremental --all-targets -- -D warnings: clean.

Performance Analysis

No runtime change: nothing calls this crate yet.

  • This merge request does not introduce any performance regression. If a performance regression is expected, explain why.
Agent context: long-form analysis, file-by-file walkthroughs, profiler output, alternatives considered

What lands

config/export.yaml         entities (source kind, id seed and columns, column sources), containment and graph edges
src/export/parse.rs        export.yaml -> compiled plan, validated against the ontology
src/export/mod.rs          export(state, lang, ontology, envelope) -> Vec<(table, RecordBatch)>; Envelope, Scalar
src/pipeline/phases.rs     Display (display rules -> tags), Export, Emit
src/templates.rs           index(): Prepare, Each(Parse+Rewrite+Canonicalize+Link), Insert, Resolve
src/shared/error.rs        Error::Export(ArrowError)
tests/export.rs

Files are copied from !2454 (closed) at their final shape; templates::reindex waits for Remap and snapshots in MR 7.

Ids

export.yaml declares each entity's id as a hash over [project_id, branch] plus a seed and named columns: file over path; def over file_path, fqn, span; import over file_path, import_path, identifier_name, span. This is the scheme code-graph uses, so the two engines produce the same id for the same row, and a reindex never changes the id of an untouched row. The test ids_depend_on_content_not_on_the_run pins it.

MR stack

MR Contents
!2620 (merged) runner, budgets, errors, canonical alphabet, artifact and graph types
!2627 (merged) Tree, Interner, tree-sitter languages, Env, inventory, Prepare + Each + Parse
!2629 (merged) DSL engine, Rewrite + Canonicalize, Env.rules, Python rules
!2631 (merged) SSA, linker, Link + Insert
!2633 (merged) resolver, project tree, Resolve
6 (this) export engine, Display + Export + Emit, index template
7 YAML suites through the integration-tests-codegraph runner with an engine switch; CI job
8 State save/load, Remap, reindex template, incremental suites
9 remaining language rule files
10 orbit index --engine incremental, full index through the existing DuckDB sink
11 snapshot next to the DuckDB file, changed paths from git, incremental CLI
12 default flip per language; old pipeline and its fixtures retire

%{all_commits}

Edited by Michael Usachenko

Merge request reports

Loading
Loading