Fix zoekt search pagination counts
What does this MR do and why?
Because recent versions of zoekt searches will return a finite amount of results (https://gitlab.com/gitlab-org/gitlab-zoekt-indexer/-/blob/e663367db56e4327827ae44815897156a4fd7538/internal/search/search.go#L160), and the number of total results does not always match the number of results displayed to the user, the pagination was incorrect. This means that AST queries had some pagination numbers with no results. The pagination assumes FileCount and MatchCount match what is being displayed to the user, not a count across the entire corpus of matches.
Historically, FileCount and MatchCount in the response were accurate because we didn't impose any limits so counts were based on the entire corpus of the search.
Until we introduce a new key to reflect the "displayed count", we will need to count the results on rails. This counting behavior is behind the feature flag zoekt_ast_search_payload.
Lastly, per the documentation (https://docs.gitlab.com/development/exact_code_search/#send-requests-directly-to-the-zoekt-webserver), this MR bumps up the match windows which were previously smaller than the number of results we are requesting.
AI Summary
This change improves how search results are counted in Zoekt (a code search engine). Previously, the system relied on global counters provided by the search service, but now it can optionally count the actual results returned instead.
The update adds a feature flag that allows switching between the old counting method (using pre-calculated totals) and the new method (counting individual files and matches in the response). When the new method is enabled, it provides more accurate counts by examining the actual search results rather than trusting summary statistics.
The change also adds some search parameters to potentially retrieve more results for better counting accuracy, passes user information through the system to enable the feature flag, and includes tests to verify both counting methods work correctly.
References
Screenshots or screen recordings
Search result counts are now the same whether zoekt_ast_search_payload is enabled or not.
How to set up and validate locally
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.