Zoekt search failed to convert query to proto

What does this MR do and why?

Fixes a bug where Zoekt code search failed to convert a query to proto when the input contained an unterminated quote followed by a filter-like token (for example "file:test or "case:no). The old parser ran the filter regex over the whole string with no awareness of quotes, so "file:test was split into a " keyword plus a file:test filter, producing an invalid payload downstream. Related to #592064.

Root cause

In ee/lib/search/zoekt/query.rb, both keyword and filters operated on the raw query:

  • query.gsub(query_matcher_regex, '').strip stripped filter tokens regardless of where they appeared.
  • query.scan(query_matcher_regex) extracted filters from anywhere in the string, including inside quotes and from the middle of a word.
  • query_matcher_regex had no left-side boundary, so testlang:ruby was split into test + lang:ruby.

Fix

  1. Segment the query before parsing filters. A new segments method tokenizes the input into alternating quoted and non-quoted runs using /"[^"]*"|"[^"]*\z|[^"]+/. The alternation matches, in order: a balanced quoted phrase, an unterminated quoted phrase running to end-of-string, or one or more non-quote characters.
  2. Only scan non-quoted segments for filters. filters now does segments.flat_map { |seg| quoted?(seg) ? [] : seg.scan(query_matcher_regex) }, so content inside "..." is never interpreted as a filter directive.
  3. Treat quoted content as a literal phrase. format_segment strips the surrounding quotes and, in :exact mode, escapes the inner content via RE2::Regexp.escape. Unterminated quotes ("file:test) are handled the same way, with only the opening quote stripped.
  4. Require a left boundary on filter tokens. query_matcher_regex is now /(?<=\A|\s)-?#{filter}:\S+/, so testlang:ruby stays a single literal keyword instead of being split.
  5. Collapse whitespace around removed filters. keyword_segments uses gsub(query_matcher_regex, ' ').squeeze(' ').strip, so foo lang:ruby bar produces the keyword foo bar rather than foo bar.

Test coverage

Added four new contexts in ee/spec/lib/search/zoekt/query_spec.rb:

  • Quoted segments containing filter-like tokens — regression coverage for "file:test, "case:no, "file:test", "-file:test", "", "foo" "bar", etc., in both :exact and :regex modes.
  • Interaction with Filters.by_query_string case-modifier guard — pins the contract that an unquoted case:no still matches /\bcase:(no|yes|auto)\b/ downstream, while a quoted "case:no" is escaped to case\:no and intentionally does not (so the downstream filter will prepend its own case:no).
  • Filter tokens without a left-side word boundarytestlang:ruby, foofile:bar, _lang:ruby, 1lang:ruby stay literal.
  • Filters in the middle of a keyword runfoo lang:ruby bar collapses to foo bar lang:ruby, including multi-space inputs.

The existing repo:foo test was rewritten to reflect the new boundary semantics (repo:foo bar f:test instead of test repo:foo).

References

Related to #592064

Screenshots or screen recordings

Before After
Screenshot_2026-06-02_at_10.54.17 Screenshot_2026-06-02_at_10.51.27

How to set up and validate locally

  • Add a code like this file:test
  • Perform a search file:test. It should should all files with test name in it.
  • Perform a search "file:test". It should perform an exact match search for file:test`.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #592064

Edited by Ravi Kumar

Merge request reports

Loading