One Parser (gitlab-code-parser)
# Vision Statement
Establish a single, efficient, and reliable static code analysis library (`gitlab-code-parser`) built in Rust. This library will serve as the foundation for diverse code intelligence features across GitLab, from server-side indexing (<a href="https://gitlab.com/groups/gitlab-org/-/epics/17514">Knowledge Graph</a>, [Embeddings](https://gitlab.com/groups/gitlab-org/-/epics/16910#note_2416426797)) to client-side analysis (Language Server, Web IDE). Initially scoped to AI and Editor Features.
{width=312 height=312}
> See the <a href="https://gitlab.com/gitlab-org/gitlab/-/issues/534153">Original Proposal</a>
# Problem to Solve
The need for advanced code understanding is growing within GitLab, powering features such as:
* **[Knowledge Graph](https://gitlab.com/groups/gitlab-org/-/epics/17514):** Requires detailed analysis of code entities (classes, functions, modules) and their relationships (calls, imports, inheritance) to build a comprehensive graph representation of codebases.
* **[Codebase Chat Context / Embeddings](https://gitlab.com/groups/gitlab-org/-/epics/16910#note_2416426797):** Needs to parse code to create meaningful chunks (e.g., by class or function) for generating embeddings used in semantic search and AI-driven chat features.
* **[Language Server / IDE Features](https://gitlab.com/gitlab-org/editor-extensions/gitlab-lsp/-/blob/main/docs/developer/tree-sitter.md):** Must analyze code (including uncommitted local changes) to provide real-time feedback, navigation, and context-aware assistance directly to the developer.
[Discussions](https://gitlab.com/gitlab-org/code-creation/repo-knowledge-grapher/-/issues/7#note_2431074007) between the Knowledge Graph and Codebase Chat Context sub-teams revealed overlapping requirements for static analysis, identifying `tree-sitter` as a core dependency for parsing across multiple languages.
Building sophisticated static code analysis capabilities in silos leads to several problems:
1. **Duplicated Effort:** Teams may build and maintain separate parsing logic, wasting engineering resources.
2. **Inconsistency:** Different parsers might produce varying results or support different language features/versions, leading to inconsistent user experiences across features.
3. **Maintenance Overhead:** Maintaining multiple parsing systems increases the long-term burden on engineering.
4. **Client-Side Challenges:** Providing features that react to local code changes (e.g., in the Web IDE or via Language Servers) requires efficient, potentially client-side parsing. Integrating server-only parsing solutions with local changes is complex and often leads to stale results.
A unified approach is needed to provide a consistent, maintainable, and performant foundation for code analysis across GitLab that can run both server-side and client-side.
# Solution
Build **"One Parser" (`gitlab-code-parser`)**: A shared static code analysis library implemented in Rust.
**Core Characteristics:**
1. **Project:** `gitlab-code-parser` ([gitlab-org/code-intelligence/gitlab-code-parser](https://gitlab.com/gitlab-org/code-creation/gitlab-code-parser) - *Note: Namespace discussed in [issue comment](https://gitlab.com/gitlab-org/gitlab/-/issues/534153#note_2439182274)*)
2. **Exports**: A Rust `crate`, WASM Bindings, and FFI bindings for other runtimes.
3. **Parsing Engine:** Leverages `tree-sitter` and `ast-grep` ([POC](https://gitlab.com/michaelangeloio/ast-grep-parser-rust)) and their respective ecosystems of language grammars/rules.
4. **Design:** A largely **stateless** library. It will accept source code content (as a string), other metadata, and the language identifier as input.
5. **Output:** Returns structured information derived from the Abstract Syntax Tree (AST), such as class/function definitions, imports/exports, method calls, rule matches, etc. The exact output structure will be defined based on the requirements of initial consumers ([Knowledge Graph](https://gitlab.com/groups/gitlab-org/-/epics/17514), Embeddings, Language Server).
6. **Versioning:** The library will be versioned to allow consuming applications to adopt updates independently.
## Core Runtimes
The library will be built with interoperability in mind, providing native bindings or interfaces for the following core runtimes:
* **Rust:** Consumable directly as a crate (`gitlab-code-parser`).
* **Node.js:** Via pre-compiled native addons using [`napi-rs`]([https://napi.rs/](https://napi.rs/)), distributed via npm.
* **Golang:** Via C Foreign Function Interface (FFI) using `cgo` and `cbindgen` ([POC](https://gitlab.com/michaelangeloio/gitlab-code-parser-go-ffi)).
* **Ruby:** Via FFI (e.g., using `ffi-rs` or similar gems), provided when demand is proven.
## Architecture
### Overview
The `gitlab-code-parser` library is designed as a central Rust core containing static analysis logic using `tree-sitter` and `ast-grep`. This core library can be compiled into different formats to be consumed by various GitLab components and runtimes. Server-side components (like indexers running in Go or Ruby) can use FFI bindings, while client-side components (like the Language Server running in Node.js or Web IDE) can use native Node addons or WASM modules. This ensures the same parsing logic is used everywhere, providing consistency and performance.
### Components
```mermaid
graph TD
subgraph gitlab-code-parser Rust Project
direction LR
Core["Core Rust Library <br/><i>(Parsing Logic, Tree-sitter/ast-grep Integration, AST Traversal, Rule Matching)</i>"]
FFI["FFI Layer <br/><i>(cbindgen, ffi-rs)</i>"]
NAPI["Node Addon Layer <br/><i>(napi-rs)</i>"]
WASM["WASM Layer <br/><i>(wasm-bindgen / wasi)</i>"]
Core -- Exports --> FFI
Core -- Exports --> NAPI
Core -- Exports --> WASM
end
subgraph Consuming Runtimes / Environments
NodeJS["Node.js <br/><i>(Language Server)</i>"]
Golang["Golang <br/><i>(Rails Indexer Backend)</i>"]
Ruby["Ruby <br/><i>(Rails Monolith)</i>"]
Browser["Browser / Web IDE <br/><i>(Client-side Analysis)</i>"]
RustApp["Other Rust Apps <br/><i>(e.g., Knowledge Graph Indexer)</i>"]
end
NAPI -- Consumed by --> NodeJS
FFI -- Consumed by --> Golang
FFI -- Consumed by --> Ruby
WASM -- Consumed by --> Browser
Core -- Consumed by --> RustApp
style Core fill:#000,stroke:#333,stroke-width:2px
style FFI fill:#000,stroke:#333,stroke-width:2px
style NAPI fill:#000,stroke:#333,stroke-width:2px
style WASM fill:#000,stroke:#333,stroke-width:2px
```
* **Core Rust Library:** The central `gitlab-code-parser` crate containing the primary logic. It integrates with parsing libraries (`tree-sitter`, `ast-grep`), loads language grammars/rules, parses input source code into an AST, and provides functions to query or extract structured information from the AST.
* **FFI Layer:** Exposes C-compatible functions and data structures from the core Rust library. Tools like `cbindgen` automatically generate C headers (`.h`) from the Rust FFI definitions, enabling integration with languages like Go (using `cgo`) and Ruby (using FFI gems). The [Go FFI POC](https://gitlab.com/michaelangeloio/gitlab-code-parser-go-ffi) demonstrates this approach.
* **Node Addon Layer:** Uses `napi-rs` to build pre-compiled Node.js native addons (`.node` files) for various platforms (macOS, Linux, Windows). This allows Node.js applications (like the GitLab Language Server) to call Rust functions directly with near-native performance and type safety, avoiding the need for WASM in environments where native addons are feasible. See [Async Update details](https://gitlab.com/gitlab-org/gitlab/-/issues/534153#note_2438704151).
* **WASM Layer:** Compiles the core Rust library (including C dependencies like `tree-sitter`) to WebAssembly (`.wasm`). This enables the parser to run directly in browsers (for the Web IDE) or other JavaScript environments where native addons are not suitable. Tools like `wasm-bindgen` can facilitate type-safe interaction between JavaScript and the WASM module.
### Runtime Support Details
* **Rust:** Consumed directly as a dependency via `Cargo.toml`. The core library itself is a Rust crate.
* **Node.js:** Supported via native addons built with `napi-rs`. The library will be packaged and published to npm with pre-compiled binaries for different target platforms, allowing easy installation (`npm install`) and usage in projects like the [GitLab Language Server](https://gitlab.com/gitlab-org/editor-extensions/gitlab-lsp). This avoids the overhead of WASM for desktop/server-side Node.js use cases.
* **Golang:** Supported via FFI and `cgo`. The Rust build process generates a static library (`.a`) and a C header file (`.h`) using `cbindgen`. Go applications can then link against this static library using `cgo` directives, allowing Go code to call the exported Rust functions. The [Go + Rust + Tree-Sitter Code Parser POC](https://gitlab.com/michaelangeloio/gitlab-code-parser-go-ffi) demonstrates this mechanism for extracting symbols.
* **Ruby:** Supported via FFI. Similar to Golang, the C-compatible FFI layer exposed by the Rust library can be accessed from Ruby using standard Ruby FFI gems (like the `ffi` gem). This allows the Rails monolith or other Ruby tools to call the Rust parser functions. Specific Ruby bindings (as a Gem) will be developed based on demand.
* **WASM (for Browsers/Web IDE):** Supported by compiling the Rust core to WebAssembly. This allows the parser to run efficiently in browser environments for features within the Web IDE, handling local code changes without server roundtrips.
## Why Rust?
Rust directly addresses the core requirements of building a performant, reliable, and versatile static analysis library suitable for both server-side and client-side deployment within GitLab.
Some of Rust's benefits include:
1. **Tree-sitter Integration:**
* `tree-sitter`, the chosen parsing framework, is fundamental to this project. Rust offers **first-class, well-maintained Tree-sitter bindings** directly provided by the `tree-sitter` organization via the official [`tree-sitter` crate](https://crates.io/crates/tree-sitter)).
* This ensures reliable access to the latest Tree-sitter features, a wide range of language grammars, and stability backed by the core maintainers, reducing dependency risk compared to less active or third-party bindings in other languages.
2. **Native Performance & Interoperability (FFI/Napi):**
* Rust provides **C/C++ level performance** without garbage collection pauses, crucial for analyzing large codebases quickly.
* Rust's excellent FFI capabilities allow seamless integration with Go (via `cgo`), Ruby (via FFI gems), and Python (via `PyO3`, if needed later).
* Frameworks like `napi-rs` enable the creation of high-performance, non-blocking native Node.js addons, ideal for the Language Server. This avoids WASM overhead where native integration is possible.
3. **WebAssembly (Wasm) Compilation:**
* A critical requirement is the ability to run the parser efficiently **client-side** (e.g., within the Web IDE) to handle local code changes.
* Rust has **robust support for compiling to Wasm**, crucially including its C dependencies (like `tree-sitter`). This allows the *exact same Rust codebase* to run natively on the server (via FFI/Napi) and directly within JavaScript/TypeScript environments (via WASM) with near-native performance.
4. **Memory Safety and Reliability:**
* Rust's compile-time memory safety guarantees (borrow checker) eliminate common errors like null pointer dereferences and buffer overflows **without runtime overhead**.
* This leads to a more reliable and secure foundational library.
5. **Ecosystem:**
* Leveraging Rust opens access to other high-performance parsing and developer tooling libraries within the Rust ecosystem, such as [`ast-grep`](https://www.google.com/search?q=%5Bhttps://ast-grep.github.io/%5D\(https://ast-grep.github.io/\)) ([POC](https://gitlab.com/michaelangeloio/ast-grep-parser-rust)), [`swc`](https://www.google.com/search?q=%5Bhttps://swc.rs/%5D\(https://swc.rs/\)), [`comrak`](https://www.google.com/search?q=%5Bhttps://crates.io/crates/comrak%5D\(https://crates.io/crates/comrak\)), and GitLab's own [`glql`](https://www.google.com/search?q=%5Bhttps://gitlab.com/gitlab-org/gitlab-query-language/glql-rust/%5D\(https://gitlab.com/gitlab-org/gitlab-query-language/glql-rust/\)), potentially unifying more analysis tasks in the future.
*(For a discussion on why Golang was not chosen primarily due to Tree-sitter C dependency and WASM compilation challenges, see the original [proposal issue](https://gitlab.com/gitlab-org/gitlab/-/issues/534153))*
epic