fix(codegen): keep emoji outside the BMP intact in generated GraphQL
What does this MR do?
Fixes GLQL label filters (and any other quoted value) containing emoji outside the Basic Multilingual Plane, such as 🌐 or 🚦.
Root cause
validate_and_format_graphql parses the generated query with graphql_parser and pretty-prints it back. graphql_parser 0.4.1 prints every character above U+FFFF as \u followed by its decimal code point (src/format.rs#L148), so 🌐 (U+1F310 = 127760) was emitted as \u127760. GraphQL reads that as \u1277 (ቷ) plus the literal digits 60:
curl -s -X POST https://gitlab.com/api/graphql -H "Content-Type: application/json" \
-d '{"query":"{ a: echo(text: \"\\u127760Network\") b: echo(text: \"🌐Network\") }"}'
# {"data":{"a":"nil says: ቷ60Network","b":"nil says: 🌐Network"}}So label = "🌐Network::❗Operational Failure" asked GitLab for the label ቷ60Network::❗Operational Failure, which does not exist. Emoji inside the BMP (⏳ U+23F3, ❗ U+2757) are printed verbatim and kept working, which explains the "hit or miss" reports in the issue.
Upstream has an open fix, graphql-rust/graphql-parser#97, but no release ships it (0.4.1 is the latest). Patching the crate is not an option either: the gem's native extension pulls glql as a git dependency, so a [patch] in this crate's manifest would not reach it, and vendoring ~4.4k lines for a one-line fix is not worth it.
Fix
AstralPlaceholders swaps each character above U+FFFF for an unused BMP Private Use Area code point before the parse/format round trip and restores it on the printed output. Stand-ins are chosen from code points absent from the query both as characters and as \uXXXX escapes, so restoring cannot touch user text or chain. Parse errors are reported against the original query so they name the user's characters. A canary unit test asserts that graphql_parser still mangles astral characters, so a future dependency bump that fixes it flags the workaround for removal.
Follow-ups
- Release glql and bump
gitlab_query_language/@gitlab/query-language-rustingitlab-org/gitlab; the monolith only gets the fix then.
Example Usage
Build prerequisite:
cd glql_rb
bundle install
bundle exec rake compileFilter by a scoped label whose emoji sit outside the BMP
mode: standard
query: group = "gitlab-org" and label = "Anne 🔍"
fields: title, labelsTest via ruby extension
cd glql_rb # after the build prerequisite above
bundle exec ruby -Ilib - <<'RUBY'
require "gitlab_query_language"
require "json"
require "net/http"
query = 'group = "gitlab-org" and label = "Anne 🔍"'
context = { fields: "title, labels" }
compiled = Glql.compile(query, context)
puts "== Generated GraphQL =="
puts compiled["output"]
uri = URI("https://gitlab.com/api/graphql")
http = Net::HTTP.new(uri.host, uri.port); http.use_ssl = true
req = Net::HTTP::Post.new(uri, "Content-Type" => "application/json")
req.body = { query: compiled["output"], variables: { limit: 3 } }.to_json
response = JSON.parse(http.request(req).body)
transformed = Glql.transform(response["data"], { mode: context[:mode], fields: compiled["fields"] })
puts "\n== Transformed rows =="
puts JSON.pretty_generate(transformed["data"])
RUBYGenerated GraphQL (truncated):
query GLQL($before: String, $after: String, $limit: Int) {
group(fullPath: "gitlab-org") {
workItems(labelName: "Anne 🔍", before: $before, after: $after, first: $limit, includeDescendants: true) {
nodes {
id
iid
title
...Before this MR the same line read workItems(labelName: "Anne \u128269", ...). Against gitlab.com the fixed query returns count: 9; the escaped form returns count: 0.
Related to gitlab#572851 (closed)