feat(orbit-cli): index and reindex with the incremental engine behind --ff=inc

What does this MR do and why?

orbit index --ff=inc indexes with code-graph-incremental and keeps its state under ~/.gitlab/orbit/var/<project>/; orbit reindex --ff=inc reparses only what git says changed since then. First run of the engine on real repositories: the GitLab monolith indexes in 35 s and a one-file change reindexes in 16 s. Two engine hot spots that showed up on it are fixed along the way.

orbit index . --ff=inc            # full; snapshots per language family + index.json
orbit reindex . --ff=inc          # changed files only; refuses without prior state
orbit index . --ff=inc --stats    # slowest files per phase

Related Issues

Relates to #1284. Follows !2656 (merged).

Testing

cargo nextest run --test cli -E 'test(incremental_reindex)': full index, edit + untracked file, commit + rename + delete + a new language, with an alias-less language (Ruby) alongside Rust. incremental_suites 261/261, suites 247/247 unchanged.

On the GitLab monolith (100k files, 6 families): index 42 s / 5.8 GB peak, 287k definitions, 1.4M edges, 210 MB of snapshots; reindex after touching app/models/project.rb: 34 s / 5.9 GB, of which the Ruby pipeline is 6 s and the search-index rebuild 16 s. orbit grep works on the result.

Performance Analysis

Nothing runs this path without --ff=inc. Two engine changes affect every language and are the reason the monolith completes:

• The file walk read every file whole on one thread to classify it: 16 s of a monolith index before anything was parsed. walk_dir_parallel lists first and classifies across cores; both the old and the incremental index use it: 16 s → 3.5 s.

• has_incoming/has_outgoing scanned every edge per node: quadratic in display. Now an edge index built once per run; display on 50k Ruby files 5 min → 1 s.

• A wildcard import (require, from x import *, import pkg.*) got an IMPORTS edge to every definition visible from its target and resolved its call sites by walking them: require 'spec_helper' in 20k specs made 11M edges. Now one edge to the target file plus one per definition the importing code refers to (calls, supertypes, decorators), found by name in the target's visible names. Ruby edges 12.5M → 1.17M, resolve 32 s → 6 s; CALLS Definition→Definition 205,521 → 205,431.

• This merge request does not introduce any performance regression. If a performance regression is expected, explain why.

Agent context: long-form analysis, file-by-file walkthroughs, profiler output, alternatives considered

State layout

~/.gitlab/orbit/var/<project_id>/
  graph.<family>.bin     one snapshot per language family (an Env is per family)
  index.json             { commit_sha, dirty: [working-tree paths that differed from it],
                           families: [...], owner: <family that records unparsed files> }

reindex computes git diff --name-status <commit_sha> HEAD ∪ git status --porcelain -uall, then re-examines every dirty path from the last run (present → changed, gone → removed). That closes the gap between "the commit indexed" and "the working tree indexed": a file indexed while untracked, then committed and renamed, is otherwise invisible to the diff. Removed paths are routed to a family by extension, unparsed ones to the owner.

Each run replaces the project's rows: clear_project, then every family is exported, untouched ones straight from their snapshot (FamilyWork::Keep). Directories and directory→directory CONTAINS edges are emitted once per run (each family's project tree covers the directories above its files). A language with a grammar but no rule file (hcl) is unparsed. The run budget per family is --budget (30 s default); per-file budgets are the crate's.

Where the time and memory go (monolith, measured)

Ruby holds 50k trees: 14.5M canonical nodes at ~80 B (node plus arena links) ≈ 1.1 GB, 6M tags in 3.7M per-node maps ≈ 0.2 GB, 1.6M edges 45 MB. Dead nodes and source text are already dropped at canonicalize. Resolve adds ~1.2 GB of transient maps. The snapshot used to add 2.1 GB on top by cloning every tree and building the whole archive in memory; it is now a framed stream, one tree at a time. Earlier families' freed memory is not returned to the OS by the allocator, so RSS carries ~1.2 GB into Ruby.

A reindex loads only the families with changes, drops their previous rows, exports them, and stamps the rest with the new commit. Of its 34 s: Ruby pipeline 6 s, snapshot load 1.5 s and save ~3 s, row deletes and stamping a few seconds, search-index rebuild 16 s (def_doc table plus DuckDB FTS, rebuilt whole by the same code the old pipeline uses; DuckDB's FTS index cannot be updated in place).

What is not in this MR

• An incremental search index: the 16 s above.
• Progress bars: phase lines on the log, JSON at the end.
• Materialising only dirty trees on load (zero-copy archive access): would take reindex memory from ~6 GB to the size of the dirty set.

Wildcard imports, the reasoning

The per-definition edges had two consumers: the export rows, and the call-site loop in resolve_one_import, which matched each call's name against every materialised target. Incremental dirtiness never depended on them (it comes from reqs and from visible locations in dirty files), and the member-call block cannot match a *. So the same CALLS come out of a by-name lookup in visible[target], and the import edges now record what the file actually refers to instead of what it could see. A qualified root such as Security::ProjectTrackedContext (Ruby's scope_resolution flattens the prefix to one symbol) resolves segment by segment. Wildcard references appear both from the linker and from wave 1's unbound names, so their uses resolve after wave 1.

MR stack

MR	Contents
!2620 (merged) … !2656 (merged)	merged: crate, rules, fixtures
this	--ff=inc index and reindex, var state
next	delta export; default flip per language

%{all_commits}