## What [Tree-sitter](https://tree-sitter.github.io/tree-sitter/) is an incremental parsing library, that can build concrete syntax tree for a source code. It utilizes a plugin paradigm, allowing many different programming languages to be parsed and analyzed using a single query interface. ## Why A parsing tool is necessary on the IDE extension layer to enhance code creation requests with additional contextual data that is only available on the client level. 1. Access all public methods of newly written modules that have not been committed just yet to the repository. 2. Improve GitLab's ability to differentiate between code completion and code generation requests based on several different factors including the cursor being in the empty function or class body etc. 3. IDEs are stateful while monolith code creation API is not, meaning that IDE will be able to reuse once constructed tree for a given source file, while monolith would need to rebuild the tree for every request ## Where The current direction is to use a language server for code suggestion features delivery. It makes sense to include tree-sitter into the language server, so that parsing capabilities are made available to all extensions. Exception: the JetBrains plugin which is not currently expected to integrate with the language server and would need to be handled separately. ## How Based on research run in https://gitlab.com/gitlab-org/gitlab/-/issues/421410#note_1560927458 usage of [WebAssembly distribution of tree-sitter](https://github.com/tree-sitter/tree-sitter/tree/master/lib/binding_web) was selected as the most promising approach. The most important advantage that WebAssembly has over other options is its platform independence, where all other options would need to be precompiled for all different platforms that are expected to support GitLab. ### MVC use case (https://gitlab.com/gitlab-org/editor-extensions/gitlab-lsp/-/issues/87) MVC: enable IDE extensions (via the Language Server) to decide on their own between making a code completion or a code generation request. This will provide a driving use-case for tree-sitter integration in the extensions. The rules to decide between code completion and code generation are currently implemented in the backend as follows: 1. If cursor is preceded by a comment that has at least 10 characters, which is followed by a line that contains either comments or only white spaces, then this request is marked as code generation. 2. Otherwise, it is treated as code completion Extensions can request a specific type of suggestion from the GitLab monolith with a new field `intent` added to the API with https://gitlab.com/gitlab-org/gitlab/-/merge_requests/132278 ### PoC For the sake of the mentioned research, a proof of concept (that uses a tree-sitter with a JavaScript plugin to extract comments from a supplied source code) was built into the VS Code desktop extension. That PoC can be viewed at https://gitlab.com/gitlab-org/gitlab-vscode-extension/-/merge_requests/1028 which might serve as a reference during later implementation stages The PoC unveiled a preliminary question that has to be addressed: The existing WebAssembly tree-sitter package is not compatible with WebIDE because it requires either a file system package or a window object to be present. It would be required to adjust the WebAssembly file loading process to be compatible with WebIDE