Treat ClickHouse and Elasticsearch timeouts as GLQL timeouts

What does this MR do and why?

GLQL treats only ActiveRecord::QueryAborted as a timeout. A ClickHouse or Elasticsearch query that runs too long falls into the generic path instead: 500, an error-tracking event, SLI error_type: other, and the per-query circuit breaker never counts it, so the same slow query keeps hitting the backend. On 2026-11-11 two in-app ClickHouse analytics requests failed this way.

This MR sends ClickHouse and Elasticsearch timeouts down the existing timeout path. It changes how GLQL classifies the failure, not the queries.

Failure Before After
ClickHouse read timeout, ClickHouse server-side TIMEOUT_EXCEEDED, Elasticsearch client timeout or 408/504 500, SLI other, breaker not counted 503 Query timed out, SLI query_aborted, breaker counted
Elasticsearch connect timeout (default typhoeus adapter) 500 503, breaker counted (see tradeoffs)
Connection failures (Net::OpenTimeout, BlockedUrlError, Gitlab::Search::Client::ConnectionError), any other ClickHouse::Client::DatabaseError 500 500 (unchanged)

Both routes, POST /api/v4/glql and the in-app Glql::BaseController route, already return 503 when timeout_occurred is set. The 503 row in doc/api/glql.md now names PostgreSQL, ClickHouse, and Elasticsearch. Its history line says 19.6: if this lands in 19.6, fold it into the line from !266611 (merged); if it slips, change it to 19.7.

Which exceptions are matched

A bare Net::ReadTimeout or Faraday::TimeoutError can come from any HTTP call during GraphQL execution, so neither is matched. Each match is raised only at the ClickHouse or Elasticsearch client boundary:

  • ClickHouse::HttpClient::ReadTimeout (new): the ClickHouse post proc re-raises every Net::ReadTimeout, including ReadTotalTimeout and HeaderReadTimeout, as this subclass, so existing rescue Net::ReadTimeout and HTTP_ERRORS rescues still match.
  • ClickHouse::Client::DatabaseError whose message starts with Code: 159. (TIMEOUT_EXCEEDED): a server-side max_execution_time hit. The match is anchored because ClickHouse error text can echo query input. GitLab sets no such limit for signed-in queries, and GitLab.com logged none for GLQL in the week to 2026-11-10, so this covers one added later.
  • Elastic::TimeoutError, added in the EE override: the Elasticsearch transport interceptor raises it for client timeouts and 408/504 responses.
Side effects
  • Every uncaught ClickHouse read timeout, Sidekiq inserts included, now reaches error tracking as ClickHouse::HttpClient::ReadTimeout and groups separately. The original class survives as cause.
  • On GLQL, a ClickHouse read timeout no longer creates an error-tracking event. Gitlab::HTTP still logs the original to exceptions_json, so there's one entry instead of two. A TIMEOUT_EXCEEDED leaves neither. An Elasticsearch timeout goes from two events to one, the interceptor's.
  • query_aborted now covers all three backends, with no new error_type. clickhouse_duration_s and elasticsearch_duration_s in the request logs tell them apart.
  • lib/click_house/http_client.rb needs a ClickHouse code owner review.

Tradeoffs

  • Decision: Elasticsearch connect timeouts count as query timeouts. With typhoeus, curl reports a connect timeout and a request timeout the same way, so both arrive as Elastic::TimeoutError and can't be told apart. The cost: if Elasticsearch is unreachable, a query that fails twice is blocked for everyone until 15 minutes after its first failure.
  • The breaker is shared. It's keyed by query SHA for all users, limit 1 per 15 minutes, as for Postgres. Two ClickHouse timeouts of the same query block it (403 in-app, 429 REST) until 15 minutes after the first one.
  • Decision: the ClickHouse guest-limit empty response stays 500. Unauthenticated selects run with a 5s cap and timeout_overflow_mode: 'break', which ClickHouse::HttpClient turns into a 504 DatabaseError. Counting it in the shared breaker would let two guest requests block the query for signed-in users, who run without the cap.
  • Agent retries. The Duo Workflow Service client retries 5xx and 429, up to 4 attempts. After this MR the second timeout trips the breaker, so the remaining attempts get a cheap 429 instead of rerunning the slow query. #639385 proposes not retrying 503 and 429.
  • Follow-up. ClickHouse time isn't charged to the GLQL DB-time budgets, which count Postgres only, so only the HTTP client timeouts bound it: https://gitlab.com/gitlab-org/gitlab/-/work_items/636811.

References

Screenshots or screen recordings

Not applicable, no UI change.

How to set up and validate locally

  1. Add raise ClickHouse::HttpClient::ReadTimeout as the first line of execute_graphql in app/services/analytics/glql/query_service.rb, then gdk restart rails-web.

  2. Run a query against a group you can read:

    curl --request POST \
      --header "PRIVATE-TOKEN: <your_access_token>" \
      --header "Content-Type: application/json" \
      --data '{"glql_yaml": "query: group = \"gitlab-org\" AND state = opened"}' \
      --url "http://gdk.test:3000/api/v4/glql"

    The first two attempts return 503 with {"error":"Query timed out"}, the third returns 429. On master, raising Net::ReadTimeout instead returns 500 every time.

  3. Swap the line for raise Elastic::TimeoutError (returns 503), then raise Net::OpenTimeout (returns 500). Change the query text each time, for example state = closed, so the breaker from step 2 doesn't apply.

  4. Remove the line and restart.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Daniele Rossetti

Merge request reports

Loading
Loading