### Summary GitLab Query Language (GLQL) is an experimental attempt to create a single query language for all of GitLab which allows for filtering and embedding content from anywhere in the product. A video demo and slides describing the language and technical approach are available in https://gitlab.com/gitlab-org/gitlab/-/issues/437935+. [`glql`](https://gitlab.com/johnhope/gitlab-query-language/-/tree/main/compiler) is also the name of the command-line utility which translates GLQL queries into other formats; such as GraphQL, URL Params, and GitLab triage syntax. Builds are available for Linux, Mac OSX and, now, WebAssembly (for use with [WASI](https://github.com/bytecodealliance/wasmtime/blob/main/docs/WASI-intro.md)). If successful, when this work is complete, any Vue component will be able to translate a GLQL query into another target format. This will allow for the fast, secure implementation of GLQL-based features across the GitLab application; such as issue filtering, embedding in Markdown, saved queries, etc. Examples of GLQL queries: * `weight = 1` * `assignee = currentUser()` * `label in ("devops::plan", "devops::create")` * `label != "backend" and author = currentUser() and weight = 1 and updated > today()` * `updated > today()` ### Proposal Bundle the new WebAssembly build with GitLab's frontend. Use the bundled library to render any `` ```glql `` code blocks into a list or table of data. Rendering of GLQL blocks should be supported behind a feature flag in the following areas: * Wikis (both group and project wikis) * Epics and epic comments * Issue and issue comments * Merge requests and merge request comments * Work items and work item comments ### Implementation To start with, the implementation steps of GLQL integration can be split into 3 stages: **First Stage** In the first stage, the **compiler** is doing most of the work, with an **executor** (on the frontend) taking the query, executing it and presenting the data: ```mermaid flowchart LR Q((glql query)) --> A(Compiler) A -->|query| E(Executor) subgraph "GitLab frontend" E --> P(Presenter) end P --> V(("View (table, list, etc)")) E --> C[(API or Datastore)] C --> E ``` In this stage the language does not support `OR` or parenthesis, and the implementation doesn't support aggregations. The presenter should support simple display options like `table`, or `list` and a setting to select what `fields` to display. The presentation options should support rendering data in a neatly formatted manner ( https://gitlab.com/gitlab-org/gitlab/-/issues/477311 ) and display GitLab references exactly as they are displayed in Markdown ( https://gitlab.com/gitlab-org/gitlab/-/issues/477310 ). This should be enough to validate the idea and approach with significant functionality for users. But it does have some limitations, like lack of support for joining two tables or subqueries (eg. fetching the last comment of an issue). **Second Stage** In the second stage the **compiler** is returning a list of queries to be executed (can be a list of 1), along with possible instructions on how to combine the results (union, intersection, sort, limit, etc). The **executor** is executing the queries and aggregating them appropriately, then presenting them as before: ```mermaid flowchart LR Q((glql query)) --> A(Compiler) A -->|"[query]"| E(Executor) subgraph "GitLab frontend" E --> Ag(Aggregator) Ag --> P(Presenter) end P --> V(("View (table, list, etc)")) E --> C[(API or Datastore)] C --> E ``` In this case `or` and parentheses are supportable. The compiler will reduce the `glql` query to its simplest form and return the queries to satisfy it. Both **compiler** and **executor** get considerable more complex, however. There are also challenges, such as building in a complexity calculator so that the number of queries being combined is manageable. For example, for a hypothetical query: `weight = null or milestone = null limit 10` can only be represented as two queries when the target API is GraphQL. The executor would execute both queries, combine the (up to 20) results and limit the total set to 10. By the way, the query: `assignee = "johnhope" or assignee = "himkp" limit 10` can be represented as a single query `assignee IN ("johnhope", "himkp") limit 10`. The compiler will always try to use the fewest queries. **Final Stage** It's hard to say where we'll land in the end. A "perfect" solution would converge on using something like elasticsearch as the main datastore with a thin layer for permissions on top (probably already built by the Global Search team but I haven't checked). ES takes lucene syntax so we would only need a new code generator in the compiler and most aggregation can be done there. The difficult part is maintaining a ES cluster with all supported fields indexed, with low ingestion times. And we would always need the GraphQL fallback option for self-hosted customers who don't use ES (about 80%). --- Note there may be (and probably should be) many stages between two and final. Stage two itself might be broken into many (e.g. start with supporting counts, group by, etc before going to `or`). #### Caveats This is very experimental work. The module may be too large and require refactoring. There may be instability or performance issues when actually using the module. However, in principle, this is quite straightforward. ### About GLQL `glql` is still in rapid development but stable builds are available. * Recent build are always available from the main branch in the project's [Artifacts](https://gitlab.com/johnhope/gitlab-query-language/-/artifacts) page. `glql` takes target and script via command line arguments like so: `glql -t graphql -s "weight = 1 and project = \"gitlab-org/gitlab\""` and returns JSON with a key `output` containing the requested format: ```graphql query { project(fullPath:"gitlab-org/gitlab") { issues(weight:1) { nodes { type id iid title createdAt webUrl labels { nodes { title } } } pageInfo { endCursor hasNextPage } } } } ``` The WASI standard allows these same command line arguments to be passed to the WebAssembly module. So GLQL queries can simply be quote-escaped and sent as arguments to the GLQL WASM module.