feat(scope): add support for multiple groups/projects to analytics sources

Daniele Rossettirequested to merge
feat/151-scope-lists into main

Compiler side of #151 (closed) (MR 1/2; MR 2 is the transform side).

What does this MR do and why?

group in (...) and project in (...) become scope filters for analytics sources. The paths compile to the descendantsScope input object on the engine field (gitlab-org/gitlab!248073, GitLab >= 19.3), so one panel can aggregate across several groups and/or projects.

Query Root descendantsScope
group = "a" group(fullPath: "a") none - unchanged
group in ("a", "b") organization groupFullPaths: ["a", "b"]
project in ("a/x", "b/y") organization projectFullPaths: ["a/x", "b/y"]
group = "a" and project in ("a/x", "a/y") group(fullPath: "a") projectFullPaths: ["a/x", "a/y"]
group in (...) and project in (...) organization both lists - union, as the backend ORs them
  • A list of one path standing alone is the scalar form, so group in ("a") compiles exactly like group = "a".
  • The organization root takes no argument; the backend resolves it to the current organization, which both /api/glql and /api/v4/glql set.
  • The scope hint (CompileContext.project / group) stays single-valued and is ignored when the query carries a list. No new rule: extract_context now sets the root itself, and resolve_scope_hint only fills an empty scope.
  • Standard sources keep taking a single path. A list on them is a dedicated error: `group` accepts a single value for work items. Lists of groups and projects are supported by analytics sources only.
  • At most 20 paths combined after case-insensitive dedup, mirroring the backend's AggregationScopeInput::MAX_SOURCES, so the caller gets a compile error rather than a GraphQL one.
  • Other combinations (project = "x" and group in (...), a kind repeated) are MutuallyExclusiveFilters, as two scalars already were.

Design notes

  • extract_context collects every scope expression before deciding, replacing the single-pass retain. The type stripping behaviour is unchanged.
  • Context keeps scope: Option<ScopeRoot> as the GraphQL root and gains descendants: ScopeDescendants { groups, projects } next to it; ScopeRoot gains a path-less Organization.
  • The issue says "add Organization to allowed_scopes()". This MR does not: allowed_scopes() is published in schema.json as the roots a caller can pass, and organization is not one (it is only reached through lists). Sources declare list support with a new SourceAnalyzer::supports_multi_scope() instead, published as multi_scope in the schema document, and ContextAnalyzer::validate_scope accepts the organization root only for those sources. It defaults to on for analytics-mode analyzers and off otherwise; a source can override either way.
  • Codegen mounts descendantsScope ahead of the regular filters via GraphQLFilters::and with a Token value, so the existing argument rendering is reused. First input-object argument in analytics codegen.
  • Every dumped query validates against the 19.4 schema (DUMP_GRAPHQL=1 cargo test && npm run test:graphql); the only inspector errors are the two pre-existing $epicId1 / $parentId1 duplicates in work-item subqueries.

Not in this MR: the transform side (organization response root, InvalidScope variant) - MR 2, !498 (merged). Merge the two together and release them in one gem/npm bump: on this MR alone group in ("a", "b") compiles, but transforming its organization-rooted response fails with Namespace does not exist or you do not have access to it. The backend's "sources are not accessible" failure is a GraphQL error that both /api/glql and /api/v4/glql already surface, so no compiler work is needed for it.

Review notes

  • group = ("a", "b") is a legal parse; it now gets `group` does not support the equals (`=`) operator for `("a", "b")`. Supported operators: is one of (`in`). rather than being routed to in (list = means AND on other fields, scope lists are OR).
  • Dedup happens when a list is collected, so group in ("gitlab-org", "GitLab-Org") is the scalar form like group in ("gitlab-org"). A bare string after in (group in "gitlab-org") is accepted the same way on list-capable sources rather than rejected with an operator error that lists in as supported.
  • When a scalar scope conflicts with a list, the error names the list that actually collides, same kind first (group = "a" with a group list reports group and group; project = "a/x" next to both lists reports project and project), not whichever list came first.
  • group in ("a") and group in ("b") and group = "a" and group in ("b") report `group` and `group` cannot be used together, the same message two group = already produced. Left as is.
  • group in () is its own error (`group in (...)` needs at least one path.) rather than falling through to the analyzer's "not a recognized field" message.
  • Follow-ups are tracked in gitlab-org#23512 (closed): the gem/npm bump and the docs land together in gitlab#628101 (closed). Nothing else is needed on the GitLab side, the feature is query syntax; wiring the dashboard filters into group in (...) belongs to the bindings work (&22615 / #154).
  • Paths are interpolated unescaped, the same convention as fullPath: "..." and every other quoted filter value today.
  • The drill-down form (group = "a" and project in (...)) does not check that the projects are under the group; the backend does and reports them as not accessible.
  • No version gate in the compiler, as for every other source. None is needed: the WASM is served by the instance it queries and /api/v4/glql compiles with that instance's gem, so the compiler is never newer than the backend it talks to. The >= 19.3 note in the docs is for readers, not a runtime check.

Example Usage

cd glql_rb
bundle install
bundle exec rake compile

Analytics data is private, so the snippets send GITLAB_TOKEN from the environment. They stop at the GraphQL response (from gitlab.com on 2026-09-07): transforming an organization-rooted response into rows is MR 2 (!498 (merged)), whose Example Usage adds that step.

Aggregate across several groups

mode: analytics
query: type = CodeSuggestion and group in ("gitlab-org", "gitlab-com") and timestamp >= -30d
dimensions: language
metrics: totalCount, acceptanceRate
sort: totalCount desc
Test via ruby extension 
cd glql_rb # after the build prerequisite above

bundle exec ruby -Ilib - <<'RUBY'
require "gitlab_query_language"
require "json"
require "net/http"

query   = 'type = CodeSuggestion and group in ("gitlab-org", "gitlab-com") and timestamp >= -30d'
context = { mode: "analytics", dimensions: "language", metrics: "totalCount, acceptanceRate", sort: "totalCount desc" }

# 1. Compile GLQL -> GraphQL (also returns the typed `fields` the transform needs).
compiled = Glql.compile(query, context)
puts "== Generated GraphQL =="
puts compiled["output"]

# 2. Execute against GitLab. Analytics data is private, so send a token.
uri  = URI("https://gitlab.com/api/graphql")
http = Net::HTTP.new(uri.host, uri.port); http.use_ssl = true
req  = Net::HTTP::Post.new(uri, "Content-Type" => "application/json")
req["PRIVATE-TOKEN"] = ENV.fetch("GITLAB_TOKEN")
req.body = { query: compiled["output"], variables: { limit: 3 } }.to_json
response = JSON.parse(http.request(req).body)

puts "\n== Response data =="
puts JSON.pretty_generate(response["data"])
RUBY
query GLQL($before: String, $after: String, $limit: Int) {
  organization {
    analytics {
      duoCodeSuggestions(descendantsScope: {groupFullPaths: ["gitlab-org", "gitlab-com"]}, timestampFrom: "2026-08-08 00:00") {
        aggregated(before: $before, after: $after, first: $limit, orderBy: [{direction: DESC, identifier: "totalCount"}]) {
          count
          nodes {
            dimensions {
              language
            }
            totalCount
            acceptanceRate
          }
        }
      }
    }
  }
}
{
  "organization": {
    "analytics": {
      "duoCodeSuggestions": {
        "aggregated": {
          "count": 13,
          "nodes": [
            { "dimensions": { "language": "ruby" }, "totalCount": 2406, "acceptanceRate": 0.04282744282744283 },
            { "dimensions": { "language": "" }, "totalCount": 2339, "acceptanceRate": 0.030821917808219176 },
            { "dimensions": { "language": "vue" }, "totalCount": 896, "acceptanceRate": 0.027901785714285716 }
          ]
        }
      }
    }
  }
}

Projects under one group

mode: analytics
query: type = CodeSuggestion and group = "gitlab-org" and project in ("gitlab-org/gitlab", "gitlab-org/gitlab-shell") and timestamp >= -30d
dimensions: language
metrics: totalCount, acceptanceRate
sort: totalCount desc
Test via ruby extension 
cd glql_rb # after the build prerequisite above

bundle exec ruby -Ilib - <<'RUBY'
require "gitlab_query_language"
require "json"
require "net/http"

query   = 'type = CodeSuggestion and group = "gitlab-org" and project in ("gitlab-org/gitlab", "gitlab-org/gitlab-shell") and timestamp >= -30d'
context = { mode: "analytics", dimensions: "language", metrics: "totalCount, acceptanceRate", sort: "totalCount desc" }

# 1. Compile GLQL -> GraphQL (also returns the typed `fields` the transform needs).
compiled = Glql.compile(query, context)
puts "== Generated GraphQL =="
puts compiled["output"]

# 2. Execute against GitLab. Analytics data is private, so send a token.
uri  = URI("https://gitlab.com/api/graphql")
http = Net::HTTP.new(uri.host, uri.port); http.use_ssl = true
req  = Net::HTTP::Post.new(uri, "Content-Type" => "application/json")
req["PRIVATE-TOKEN"] = ENV.fetch("GITLAB_TOKEN")
req.body = { query: compiled["output"], variables: { limit: 3 } }.to_json
response = JSON.parse(http.request(req).body)

puts "\n== Response data =="
puts JSON.pretty_generate(response["data"])
RUBY
query GLQL($before: String, $after: String, $limit: Int) {
  group(fullPath: "gitlab-org") {
    analytics {
      duoCodeSuggestions(descendantsScope: {projectFullPaths: ["gitlab-org/gitlab", "gitlab-org/gitlab-shell"]}, timestampFrom: "2026-08-08 00:00") {
        aggregated(before: $before, after: $after, first: $limit, orderBy: [{direction: DESC, identifier: "totalCount"}]) {
          count
          nodes {
            dimensions {
              language
            }
            totalCount
            acceptanceRate
          }
        }
      }
    }
  }
}
{
  "group": {
    "analytics": {
      "duoCodeSuggestions": {
        "aggregated": {
          "count": 5,
          "nodes": [
            { "dimensions": { "language": "ruby" }, "totalCount": 1648, "acceptanceRate": 0.0491504854368932 },
            { "dimensions": { "language": "vue" }, "totalCount": 796, "acceptanceRate": 0.026381909547738693 },
            { "dimensions": { "language": "js" }, "totalCount": 330, "acceptanceRate": 0.048484848484848485 }
          ]
        }
      }
    }
  }
}

Several groups, one row per project

The scope lists say which groups and projects to read; a project dimension then breaks the aggregate down per project across all of them. AgentPlatformSession and Pipeline are the two analytics sources with a project dimension (none has a group dimension).

mode: analytics
query: type = AgentPlatformSession and group in ("gitlab-org", "gitlab-com") and created >= -30d
dimensions: project
metrics: totalCount, usersCount
sort: totalCount desc
Test via ruby extension 
cd glql_rb # after the build prerequisite above

bundle exec ruby -Ilib - <<'RUBY'
require "gitlab_query_language"
require "json"
require "net/http"

query   = 'type = AgentPlatformSession and group in ("gitlab-org", "gitlab-com") and created >= -30d'
context = { mode: "analytics", dimensions: "project", metrics: "totalCount, usersCount", sort: "totalCount desc" }

# 1. Compile GLQL -> GraphQL (also returns the typed `fields` the transform needs).
compiled = Glql.compile(query, context)
puts "== Generated GraphQL =="
puts compiled["output"]

# 2. Execute against GitLab. Analytics data is private, so send a token.
uri  = URI("https://gitlab.com/api/graphql")
http = Net::HTTP.new(uri.host, uri.port); http.use_ssl = true
req  = Net::HTTP::Post.new(uri, "Content-Type" => "application/json")
req["PRIVATE-TOKEN"] = ENV.fetch("GITLAB_TOKEN")
req.body = { query: compiled["output"], variables: { limit: 3 } }.to_json
response = JSON.parse(http.request(req).body)

puts "\n== Response data =="
puts JSON.pretty_generate(response["data"])
RUBY
query GLQL($before: String, $after: String, $limit: Int) {
  organization {
    analytics {
      agentPlatformSessions(descendantsScope: {groupFullPaths: ["gitlab-org", "gitlab-com"]}, createdEventAtFrom: "2026-08-08 00:00") {
        aggregated(before: $before, after: $after, first: $limit, orderBy: [{direction: DESC, identifier: "totalCount"}]) {
          count
          nodes {
            dimensions {
              project {
                id
                name
                avatarUrl
                fullPath
                webPath
                nameWithNamespace
              }
            }
            totalCount
            usersCount
          }
        }
      }
    }
  }
}

Response (project object truncated to fullPath):

{
  "organization": {
    "analytics": {
      "agentPlatformSessions": {
        "aggregated": {
          "count": 1199,
          "nodes": [
            { "dimensions": { "project": { "fullPath": "gitlab-org/editor-extensions/gitlab-lsp" } }, "totalCount": 52453, "usersCount": 52 },
            { "dimensions": { "project": { "fullPath": "gitlab-org/ops/artifact-registry" } }, "totalCount": 25392, "usersCount": 45 },
            { "dimensions": { "project": { "fullPath": "gitlab-org/gitlab" } }, "totalCount": 24684, "usersCount": 799 }
          ]
        }
      }
    }
  }
}

Also verified live

  • Pipelines across the same two groups with dimensions: project and metrics: totalCount, successRate - 14180 projects, top row gitlab-org/gitlab with 46213 pipelines at 0.858 success rate; rows from both gitlab-org and gitlab-com appear.
  • Union of groups and a project: group in ("gitlab-org/analytics-section", "gitlab-com") and project in ("gitlab-org/gitlab") - 11 rows under the organization root.
  • Another engine: type = Pipeline and group in ("gitlab-org", "gitlab-com") and finished >= -7d with dimensions: status - 5 rows (success 206885, failed 18501, ...).
  • Unknown path: group in ("gitlab-org", "no-such-group-glql-151") - GraphQL error The following sources are not accessible: no-such-group-glql-151 with the engine field nulled, as the backend documents.
  • The first cold duoCodeSuggestions query across gitlab-org and gitlab-com returned HTTP 500 after 16s; the retry and every variant since answered in under a second, so it looks like a backend cold start rather than anything in the query.

Standard mode keeps a single scope

query: type = Issue and group in ("a", "b")
fields: title
Error: `group` accepts a single value for work items. Lists of groups and projects are supported by analytics sources only.

Checklist

  • cargo test (33 suites), cargo clippy --all-targets -- -D warnings, cargo fmt --check
  • cargo run --bin generate-schema && npm run lint:prettier:fix (adds multi_scope per source/mode)
  • DUMP_GRAPHQL=1 cargo test && npm run test:graphql - new queries valid against the 19.4 schema dump
  • Live run against gitlab.com through the Ruby extension: organization root (two groups), drill-down, union, pipelines engine, and the not-accessible error (see Example Usage)
Edited by Daniele Rossetti

Merge request reports