Treat ClickHouse and Elasticsearch timeouts as GLQL timeouts
What does this MR do and why?
GLQL treats only ActiveRecord::QueryAborted as a timeout. A ClickHouse or Elasticsearch query that runs too long falls into the generic path instead: 500, an error-tracking event, SLI error_type: other, and the per-query circuit breaker never counts it, so the same slow query keeps hitting the backend. On 2026-11-11 two in-app ClickHouse analytics requests failed this way.
This MR sends ClickHouse and Elasticsearch timeouts down the existing timeout path. It changes how GLQL classifies the failure, not the queries.
| Failure | Before | After |
|---|---|---|
ClickHouse read timeout, ClickHouse server-side TIMEOUT_EXCEEDED, Elasticsearch client timeout or 408/504 |
500, SLI other, breaker not counted |
503 Query timed out, SLI query_aborted, breaker counted |
Elasticsearch connect timeout (default typhoeus adapter) |
500 |
503, breaker counted (see tradeoffs) |
Connection failures (Net::OpenTimeout, BlockedUrlError, Gitlab::Search::Client::ConnectionError), any other ClickHouse::Client::DatabaseError |
500 |
500 (unchanged) |
Both routes, POST /api/v4/glql and the in-app Glql::BaseController route, already return 503 when timeout_occurred is set. The 503 row in doc/api/glql.md now names PostgreSQL, ClickHouse, and Elasticsearch. Its history line says 19.6: if this lands in 19.6, fold it into the line from !266611 (merged); if it slips, change it to 19.7.
Which exceptions are matched
A bare Net::ReadTimeout or Faraday::TimeoutError can come from any HTTP call during GraphQL execution, so neither is matched. Each match is raised only at the ClickHouse or Elasticsearch client boundary:
ClickHouse::HttpClient::ReadTimeout(new): the ClickHouse post proc re-raises everyNet::ReadTimeout, includingReadTotalTimeoutandHeaderReadTimeout, as this subclass, so existingrescue Net::ReadTimeoutandHTTP_ERRORSrescues still match.ClickHouse::Client::DatabaseErrorwhose message starts withCode: 159.(TIMEOUT_EXCEEDED): a server-sidemax_execution_timehit. The match is anchored because ClickHouse error text can echo query input. GitLab sets no such limit for signed-in queries, and GitLab.com logged none for GLQL in the week to 2026-11-10, so this covers one added later.Elastic::TimeoutError, added in the EE override: the Elasticsearch transport interceptor raises it for client timeouts and408/504responses.
Side effects
- Every uncaught ClickHouse read timeout, Sidekiq inserts included, now reaches error tracking as
ClickHouse::HttpClient::ReadTimeoutand groups separately. The original class survives ascause. - On GLQL, a ClickHouse read timeout no longer creates an error-tracking event.
Gitlab::HTTPstill logs the original toexceptions_json, so there's one entry instead of two. ATIMEOUT_EXCEEDEDleaves neither. An Elasticsearch timeout goes from two events to one, the interceptor's. query_abortednow covers all three backends, with no newerror_type.clickhouse_duration_sandelasticsearch_duration_sin the request logs tell them apart.lib/click_house/http_client.rbneeds a ClickHouse code owner review.
Tradeoffs
- Decision: Elasticsearch connect timeouts count as query timeouts. With
typhoeus, curl reports a connect timeout and a request timeout the same way, so both arrive asElastic::TimeoutErrorand can't be told apart. The cost: if Elasticsearch is unreachable, a query that fails twice is blocked for everyone until 15 minutes after its first failure. - The breaker is shared. It's keyed by query SHA for all users, limit 1 per 15 minutes, as for Postgres. Two ClickHouse timeouts of the same query block it (
403in-app,429REST) until 15 minutes after the first one. - Decision: the ClickHouse guest-limit empty response stays
500. Unauthenticated selects run with a 5s cap andtimeout_overflow_mode: 'break', whichClickHouse::HttpClientturns into a504DatabaseError. Counting it in the shared breaker would let two guest requests block the query for signed-in users, who run without the cap. - Agent retries. The Duo Workflow Service client retries
5xxand429, up to 4 attempts. After this MR the second timeout trips the breaker, so the remaining attempts get a cheap429instead of rerunning the slow query. #639385 proposes not retrying503and429. - Follow-up. ClickHouse time isn't charged to the GLQL DB-time budgets, which count Postgres only, so only the HTTP client timeouts bound it: https://gitlab.com/gitlab-org/gitlab/-/work_items/636811.
References
- Related: #639385
- Follow-up: https://gitlab.com/gitlab-org/gitlab/-/work_items/636811
- Same mapping for Postgres timeouts on the REST route: !266611 (merged)
Screenshots or screen recordings
Not applicable, no UI change.
How to set up and validate locally
-
Add
raise ClickHouse::HttpClient::ReadTimeoutas the first line ofexecute_graphqlinapp/services/analytics/glql/query_service.rb, thengdk restart rails-web. -
Run a query against a group you can read:
curl --request POST \ --header "PRIVATE-TOKEN: <your_access_token>" \ --header "Content-Type: application/json" \ --data '{"glql_yaml": "query: group = \"gitlab-org\" AND state = opened"}' \ --url "http://gdk.test:3000/api/v4/glql"The first two attempts return
503with{"error":"Query timed out"}, the third returns429. Onmaster, raisingNet::ReadTimeoutinstead returns500every time. -
Swap the line for
raise Elastic::TimeoutError(returns503), thenraise Net::OpenTimeout(returns500). Change the query text each time, for examplestate = closed, so the breaker from step 2 doesn't apply. -
Remove the line and restart.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.