Create a new API endpoint to run GLQL queries

Problem to solve

For MVC for Data Analyst Agent (&19500) we need Duo to be able to execute GLQL queries so that it can analyze and process the results.

The problem is that the GLQL compiler is a Typescript library that runs in the client browser (GLQL queries are compiled to GraphQL and then executed in the frontend) , and there is not a straightforward way for Duo to extract the results from the client. In addition, this also prevents supporting Duo sessions as those are executed in the background outside of the browser.

One way to solve this is to add a new GLQL API to Rails, which accepts GLQL queries, compiles them to GraphQL using the GLQL compiler, executes the resulting GraphQL query and return the data back to the consumer (in this case Duo)

The decision to be made is how to run the Typescript GLQL compiler in Rails.

Current State

  • GLQL compiler exists as a TypeScript (experimental) and Rust library in https://gitlab.com/gitlab-org/glql
    • Note: GLQL library started as a Rust library and was recently (~1mo ago) ported to Typescript (experiment running in production). Rust version is still existing but disabled (#562607).
  • GLQL queries are currently compiled to GraphQL in the browser frontend
  • Duo Chat needs to execute GLQL queries but cannot access browser-based libraries
  • This limitation prevents Duo from analyzing GLQL results in both web chat and IDE environments

Requirements:

Functional Requirements

  • Reuse existing GLQL compiler: Leverage the cript GLQL compiler from https://gitlab.com/gitlab-org/glql
  • Environment independence: Enable Duo to execute GLQL queries independently of the environment (web chat or IDE)
  • Query compilation: Accept GLQL queries and compile them to valid GraphQL
  • Query execution: Execute the compiled GraphQL queries against GitLab's GraphQL API
  • Result formatting: Return query results in a format consumable by Duo (Markdown, csv, or simplified json)
  • Error handling: Provide meaningful error messages for invalid GLQL syntax or execution failures
  • Logging: Logging: Track the GraphQL complexity score and timeouts so that we can get a sense of what Duo is generating and whether it needs any optimization.

Non-Functional Requirements

  • Performance: Query compilation and execution should complete within reasonable time limits
  • Security: Since GLQL queries use our existing security/permissions checks within Rails as it just transpiles GLQL syntax → GraphQL, the only security/permissions checks we need for this API are:
    1. Verify the API user has the right package to use GLQL (should be fine as GLQL is available on all tiers).
    2. Verify API request is genuine GLQL before trying to run it (prevent injection attempts).
  • Maintainability: Solution should be maintainable and not introduce complex dependencies

Technical Constraints

  • Must integrate with existing Rails application architecture
  • Should not duplicate GLQL compiler logic
  • Must respect existing GraphQL API permissions and rate limits

High-level API Design

POST /api/v4/glql/query

Request Format

{
  "query": "string",           // GLQL query string
  "variables": {},             // Optional query variables
  "context": {                 // Optional execution context
      //TBC
  }
}

Response Format

{
  "data": {},                  // Query results. This could either be the raw GraphQL query result or a processed version of that
  "errors": []                 // Any compilation or execution errors
}

(Note this is just a rough proposal, where the main focus is highlighting having GLQL query as input, and GraphQL - raw or processed - data as output)

Proposed Solutions

1. Interact with Rust GLQL compiler through FFI and bundle it as a gem

Wrap GLQL in a gem that links to Rust using a C dylib. We will ship precompiled gems to supported platforms, and ship Rust to compile manually for others. GLFM markdown already does this so we have a precedent set already.

Pros

  • Native performance with minimal overhead
  • Direct in-process execution without external binaries
  • Memory-safe execution environment
  • Cross-platform compatibility
  • Established Ruby FFI ecosystem and tooling
  • Established pattern in GitLab ( markdown editor )

Cons

  • Rust GLQL compiler was decommissioned due to collaboration difficulties (Requires Rust expertise within the team for maintenance)
    • (more details in the migration issue #562607)
    • If we switch back to the Rust version, we should find ways of making it easier to onboard new data sources for teams that don't have expertise in Rust.
  • The Typescript version of the compiler slightly diverted from the Rust one. We need to backport all changes made in the Typescript version to Rust since the port.
  • GitLab Omnibus integration challenges: Rust compilation dependencies, cross-platform builds
    • GLQL is on Rust 1.89 but the monolith is on 1.73 (a two year old version). Rust is needed to compile to binary for platforms where we don't support shipping a pre-compiled lib
    • Note: for MVC purposes we can add support for self-managed later.
  • FFI debugging complexity

2. Compile Typescript library to Standalone binary (PoC: glql!283 (closed))

Rails API Endpoint → Standalone GLQL Binary → GraphQL Compilation → GraphQL API → Results

Compile the TypeScript GLQL library to a standalone executable using bun (doc), which runs a full and performant JavaScript engine (JavaScriptCore) under the hood and supports cross-platform compilation for different deployment environments. The compiled binary can be bundled within a Ruby gem for simplified distribution and version management (see https://gitlab.com/gitlab-org/glql/-/jobs/11651177917/artifacts/browse/gem/pkg/).

Advantages

  • Reuses existing GLQL TS compiler
  • bun provides a fast and performant JS runtime
  • Standalone binary is easy to deploy and manage
  • Cross-platform compatibility
  • No Ruby/JavaScript interop complexity
  • Bundling as a gem provides version management, dependency resolution, and simplified distribution through RubyGems

Considerations

  • Additional binary dependency in deployment
  • Need to manage binary lifecycle and updates
  • Inter-process communication overhead
  • Security considerations: binary execution requires careful input validation and sandboxing. Considering we own all the code being run/executed though, this is a matter of safely validate/sanitise incoming queries
  • GitLab Omnibus integration challenges: binary distribution, platform-specific builds, and package management complexity

Discussion

Discussion: #573055 (comment 2814061643)

Quick items:

  • We do not ship any JS runtime today.
  • Compiling a JS runtime, and then shipping that within a Ruby Gem is ... going to complicate the security landscape. ⚠️
    • This is a massive impact, and needs approval from AppSec/ProdSec, and analysis for maintenance overhead including backports for security releases.
  • Any such called binary must support all operating platforms of the Ruby backend.
    • Consider how this would be compiled and managed in CI, and the impact on pipelines for the company.
  • Any such binary must be performant, and have very clear memory / cpu impacts documented and understood.
  • Any binaries should be as small as possible.
    • This is not a .com concern, this is impactful for every single customer on the planet.
  • bun is MIT. javascriptcore is LGPL-2. Legal will need to be consulted.

Conclusion:

it does not seem viable at the moment to support this solution, as the effort required stretches outside the team resources.

3. Run in Embedded JavaScript Engine

Execute js with a gem packaged V8 engine like mini_racer (PoC: glql!287 (closed))

Pros

  • In-process execution with no external binaries or services
  • Direct Ruby/JavaScript interop through mini_racer API
  • Simplified deployment - no additional dependencies beyond the gem

Cons

  • Critical limitation: Lack of support for asynchronous JavaScript code
  • Limited JavaScript runtime environment
  • Complex debugging across Ruby/JavaScript boundary
  • Dependency on native V8 compilation during gem installation
  • Limited access to Node.js ecosystem and modules

Discussion

Similar consideration and conclusion to the solution above, related to shipping a JS runtime.

Other alternatives we discussed:

  • Discard GLQL and let Duo generates GraphQL.
    • GraphQL scope is much larger than GLQL and chances of hallucinations or wrong answers higher
    • Requires adding the full GraphQL schema to Duo context or system prompt
    • Giving full access to the GraphQL API to an AI agent probably comes with severe security concerns
  • Rewrite GLQL in Ruby and fully port it to the backend
    • Losing the frontend GLQL library means it might be harder to support things like autocompletion through a LSP
    • Migration effort is significant and will push the MVC timeline to further ahead.
    • It might be a solution that we will consider later on for the long run.

Proof of Concepts

Next Steps

Review and Validation

  • API Team Review: Present proposed API design and implementation approach for feedback on endpoint structure, request/response formats, and integration patterns
  • Application Security (AppSec) Review: Security assessment of the standalone binary approach, input validation requirements, and potential attack vectors
  • GitLab Omnibus Review: Evaluate packaging, distribution, and deployment considerations for the standalone binary within Omnibus installations
Edited by Robert Hunt