feat(code-graph-incremental): parse phase (!2627) · Merge requests · GitLab.org / orbit / GitLab Orbit

What does this MR do and why?

Second slice of code-graph-incremental (after !2620 (merged)). A run can now walk a repository with production's file classification, read the files it has a grammar for, and parse each one into a tree, in parallel and under the run budget. Twenty languages are configured; a file parses only when production's classification marks it for parsing. Nothing is wired to the CLI or server yet.

Relates to #1284. Reference implementation: !2454 (closed).

Testing

cargo test -p code-graph-incremental: 13 tests. New in tests/parse.rs: every configured language parses its own files when classified for parsing (Haskell and OCaml are configured but production never classifies them, and the test says so); TSX inside the TypeScript pipeline gets the TSX grammar (no error nodes on JSX); files without a grammar are listed, not parsed; classifying a change set agrees with a full walk.

cargo clippy -p code-graph-incremental --all-targets -- -D warnings: clean.

Performance Analysis

No runtime change: nothing calls this crate yet.

This merge request does not introduce any performance regression. If a performance regression is expected, explain why.

Agent context: long-form analysis, file-by-file walkthroughs, profiler output, alternatives considered

What lands

config/languages.yaml        extensions, aliases, grammar, fqn separator, index names per language
src/treesitter.rs            SupportLang, LangEntry, parse(): tree-sitter CST -> Tree
src/tree/types.rs            Tree impl: append, tags, prune, compact
src/tree/walk.rs             Cursor, Walk, Step; descendants, children_of, members_by_level, reachable
src/tree/display.rs          pretty_print
src/intern.rs                Interner and Lang impls (kinds, fields, syms)
src/inventory.rs             code_filter(), walk(), classify(), timeout(): production's CodeFilter
src/env.rs                   Env { lang, lang_id, limits }; for_lang, with_limits, with_lang
src/shared/canonical.rs      the four Cursor-based helpers (def_type_of, has_def_type, ...)
src/shared/tags.rs           ReservedTags::new
src/resolver.rs              CLASS_LIKE, Resolver::new
src/pipeline/state.rs        State::new
src/pipeline/phases.rs       Prepare, Each (parallel ItemPhase runner), IntoParallel, Parse
tests/parse.rs

Files are copied from !2454 (closed) at their final shape. Two things are held back for MR 3 because they read the DSL rules: treesitter::lang_yaml (it embeds the rule files) and the manifest branch of workset (manifest names come from Env.rules). Until then every non-parse file lists as a plain file. Env grows a rules field in MR 3.

#[allow(dead_code)] remains on Tree::{node_mut, replace, postorder, preorder} (used by the rewrite engine, MR 3), Interner, Resolver and FileIndex fields (MR 5), and Listed (read by Insert, MR 4). Each attribute goes when its reader lands.

LangEntry::extensions() is the one addition over the reference: all_languages() already returned the entries, but hid their extensions, and the per-language test needs one.

Grammar choice per file

Parse picks the grammar the file's own extension names when that language belongs to the current pipeline, else the pipeline's language. So .tsx inside the TypeScript pipeline uses the TSX grammar; JavaScript also uses the TSX grammar (handles JSX).

MR stack

MR	Contents
!2620 (merged)	runner, budgets, errors, canonical alphabet, artifact and graph types
2 (this)	Tree, Interner, tree-sitter languages, Env, inventory, Prepare + Each + Parse
3	DSL (parser, rules, matching, transform, rewrite), language YAML, Rewrite + Canonicalize
4	linker, SSA, file tree, Link + Insert
5	resolver, Resolve
6	export engine + export.yaml, Display + Export + Emit, index template; YAML suites through the integration-tests-codegraph runner with an engine switch; CI job
7	State save/load, Remap, reindex template, incremental suites
8	`orbit index --engine incremental`, full index through the existing DuckDB sink
9	snapshot next to the DuckDB file, changed paths from git, incremental CLI
10	default flip per language; old pipeline and its fixtures retire

%{all_commits}

feat(code-graph-incremental): parse phase