Zoekt search failed to convert query to proto
What does this MR do and why?
Fixes a bug where Zoekt code search failed to convert a query to proto when the input contained an unterminated quote followed by a filter-like token (for example "file:test or "case:no). The old parser ran the filter regex over the whole string with no awareness of quotes, so "file:test was split into a " keyword plus a file:test filter, producing an invalid payload downstream. Related to #592064.
Root cause
In ee/lib/search/zoekt/query.rb, both keyword and filters operated on the raw query:
query.gsub(query_matcher_regex, '').stripstripped filter tokens regardless of where they appeared.query.scan(query_matcher_regex)extracted filters from anywhere in the string, including inside quotes and from the middle of a word.query_matcher_regexhad no left-side boundary, sotestlang:rubywas split intotest+lang:ruby.
Fix
- Segment the query before parsing filters. A new
segmentsmethod tokenizes the input into alternating quoted and non-quoted runs using/"[^"]*"|"[^"]*\z|[^"]+/. The alternation matches, in order: a balanced quoted phrase, an unterminated quoted phrase running to end-of-string, or one or more non-quote characters. - Only scan non-quoted segments for filters.
filtersnow doessegments.flat_map { |seg| quoted?(seg) ? [] : seg.scan(query_matcher_regex) }, so content inside"..."is never interpreted as a filter directive. - Treat quoted content as a literal phrase.
format_segmentstrips the surrounding quotes and, in:exactmode, escapes the inner content viaRE2::Regexp.escape. Unterminated quotes ("file:test) are handled the same way, with only the opening quote stripped. - Require a left boundary on filter tokens.
query_matcher_regexis now/(?<=\A|\s)-?#{filter}:\S+/, sotestlang:rubystays a single literal keyword instead of being split. - Collapse whitespace around removed filters.
keyword_segmentsusesgsub(query_matcher_regex, ' ').squeeze(' ').strip, sofoo lang:ruby barproduces the keywordfoo barrather thanfoo bar.
Test coverage
Added four new contexts in ee/spec/lib/search/zoekt/query_spec.rb:
- Quoted segments containing filter-like tokens — regression coverage for
"file:test,"case:no,"file:test","-file:test","","foo" "bar", etc., in both:exactand:regexmodes. - Interaction with
Filters.by_query_stringcase-modifier guard — pins the contract that an unquotedcase:nostill matches/\bcase:(no|yes|auto)\b/downstream, while a quoted"case:no"is escaped tocase\:noand intentionally does not (so the downstream filter will prepend its owncase:no). - Filter tokens without a left-side word boundary —
testlang:ruby,foofile:bar,_lang:ruby,1lang:rubystay literal. - Filters in the middle of a keyword run —
foo lang:ruby barcollapses tofoo bar lang:ruby, including multi-space inputs.
The existing repo:foo test was rewritten to reflect the new boundary semantics (repo:foo bar f:test instead of test repo:foo).
References
Related to #592064
Screenshots or screen recordings
| Before | After |
|---|---|
![]() |
![]() |
How to set up and validate locally
- Add a code like this
file:test - Perform a search
file:test. It should should all files withtestname in it. - Perform a search
"file:test". It should perform an exact match search forfile:test`.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
Related to #592064

