### Summary Advanced Search requests that go through `Gitlab::Search::Client.execute_search` return HTTP 200 with a silently truncated result set when Elasticsearch answers `timed_out: true`. Requests on the legacy `Elasticsearch::Model` path do not: a monkey patch raises `Elastic::TimeoutError` on the same response, so the caller gets a 408 or a 500 instead of a wrong number. The `timed_out` check lives in one place, `ee/config/initializers/elastic_client_setup.rb:45`, inside an override of `Elasticsearch::Model::Searching::SearchRequest#execute!`. `Gitlab::Search::Client#execute_search` (`ee/lib/gitlab/search/client.rb:90-99`) calls the transport directly and never passes through it, so every scope migrated to that client lost the check. `Search::Elastic::ResponseMapper` exposes `#failed?` off `response['error']` only (`ee/lib/search/elastic/response_mapper.rb:47-53`), and a shard timeout carries no `error` key. All 11 `execute_search` call sites on `master` are affected — none reads `timed_out` or `_shards.failed`: ``` ee/lib/search/elastic/relation.rb ee/lib/gitlab/elastic/search_results.rb ee/lib/ee/search/scope_handlers/groups.rb ee/lib/search/advanced_finders/sbom/aggregations_finder.rb ee/lib/search/advanced_finders/security/vulnerability/{count_by_age,count_by_severity, count_over_time,identifier_names,risk_scores,top_cwes}_finder.rb ``` ### Steps to reproduce 1. Issue a group-scoped Advanced Search for a scope served by `Gitlab::Search::Client` — `work_items`, `issues` or `epics` (`ee/lib/gitlab/elastic/search_results.rb:502`, `:546`) — against a namespace large enough for the search to exceed the 30s `timeout` set at `ee/lib/gitlab/search/client.rb:92`. 2. Observe the response: HTTP 200, a `total_count` computed from the shards that answered in time. 3. Repeat with a scope still on the legacy path — `milestones`, `notes`, `merge_requests`, `projects` (`ee/lib/gitlab/elastic/search_results.rb:479-485`). 4. Observe HTTP 408 (`SearchController#count`) or 500 (`/api/v4/groups/:id/search`), raised from `elastic_client_setup.rb:45`. ### What is the current *bug* behavior? A partial answer is indistinguishable from a complete one. `total_count` is read straight off `hits.total.value` (`response_mapper.rb:31-33`), so a timed-out search reports a smaller number as fact — no exception, no log field a caller can branch on, no flag on the response. The scope split is visible in production. Within a single controller action over 3 days, holding the timeout condition fixed (`elasticsearch_timed_out_count >= 1`, `SearchController#count`): | scope | client | HTTP 200 | HTTP 408 | |---|---|---|---| | `work_items` | `Gitlab::Search::Client` | **99** | 0 | | `milestones` | `Elasticsearch::Model` | 0 | 257 | | `notes` | `Elasticsearch::Model` | 0 | 153 | | `merge_requests` | `Elasticsearch::Model` | 0 | 12 | | `projects` | `Elasticsearch::Model` | 0 | 10 | | `commits` | `Elasticsearch::Model` | 0 | 4 | Zero crossover in either direction: the client a scope uses fully determines whether a timed-out search fails closed or returns a wrong count. The same 3-day window also shows 3 partial `GraphqlController#execute:groupVulnerabilitiesOverTime` responses, from `count_over_time_finder.rb:26`. `/api/v4/groups/:id/search` is **not** affected today. Its 215 timed-out requests in the same window are all `scope=blobs`, which reaches Elasticsearch through `RepositoryClassProxy` on the legacy path, and all 215 raise `Elastic::TimeoutError` from `elastic_client_setup.rb:45` (p90 59.3s, HTTP 500). That endpoint fails loudly and slowly — a separate problem, but not a correctness one. The exposure arrives when its scopes migrate. ### What is the expected *correct* behavior? A timed-out or shard-failed search is not silently presentable as a result. `Gitlab::Search::Client#execute_search` should treat `timed_out: true` the way `elastic_client_setup.rb:45` does, so behaviour does not depend on which client a scope happens to use; or each caller should be able to ask the response whether it is complete and decide — raise, or report the count as unknown rather than as a small number. ### Relevant logs and/or screenshots From the gprd Rails log, 3-day window, `json.elasticsearch_timed_out_count >= 1`: ``` status 200 : 102 (99 work_items + 3 groupVulnerabilitiesOverTime) <- partial, silent status 408 : 436 (legacy-path scopes, SearchController#count) status 500 : 220 (215 blobs via /api/v4/groups/:id/search + 5 others) ``` All 220 of the 500s carry `exception.class = Elastic::TimeoutError` with `elastic_client_setup.rb:45` at the top of the backtrace; 9 further `Elastic::TimeoutError` on the same endpoint carry `elasticsearch_timed_out_count = 0` and come from the Faraday transport timeout at `lib/gitlab/instrumentation/elasticsearch_transport.rb:21-24`, which is a different failure. Every affected request has `elasticsearch_calls = 1` and `elasticsearch_timed_out_count = 1`, so this is one search giving up rather than one of several. ### Output of checks This bug happens on GitLab.com. ### Possible fixes Add the completeness check to `Search::Elastic::ResponseMapper`, next to `#failed?`, covering both ways Elasticsearch reports an incomplete answer at HTTP 200 — `timed_out` and `_shards.failed > 0` — and have `Gitlab::Search::Client#execute_search` apply it, so a newly migrated scope inherits the behaviour instead of having to remember it. `#failed?` reading `response['error']` is correct for its own case and should stay; the gap is that a shard timeout is not an error response. Note also that `#error_type` calls `dig` on that key (`response_mapper.rb:55-57`), which raises if a proxy leaves a non-Hash body there. Open question: should the count callers raise, or report the count as unknown? Raising matches the legacy path and the existing `SearchController` `rescue_from`; degrading to an unknown count matches what the PostgreSQL `fast_fail` timeout already does for the same counts, and does not turn a slow namespace's tab counts into an error page.