Return 503 and 500 from the GLQL API for server failures
What does this MR do and why?
POST /api/v4/glql answers every execution failure with 400 Bad Request, including statement timeouts and unexpected exceptions. Clients, including the Data Analyst agent, are told their query was invalid when the server failed. The failures also never reach error tracking or the Rails request error SLI, and the Kibana "GLQL query validity rate" counts slow queries and bugs as the agent writing invalid GLQL.
This MR handles these cases the way the UI's GraphQL route (Glql::BaseController) already does:
| Failure | Before | After |
|---|---|---|
Statement timeout (ActiveRecord::QueryAborted) |
400, Query timed out |
503, Query timed out |
| Unexpected exception during execution | 400 with the exception message |
re-raised, so the API's normal exception handling applies: 500 plus error tracking, unless Grape has a more specific handler (for example ActiveRecord::RecordNotFound becomes 404) |
| YAML parse, compile, GraphQL validation or field, and transform errors | 400 |
400 (unchanged) |
Group filter over a finder limit (TooManyGroupMembersError, TooManyAssignedIssuesError) |
400 with the limit message |
400 with the limit message (unchanged, through a rescue_from in API::Glql) |
Unusable after cursor (Gitlab::Pagination::Keyset::InvalidCursorError) |
400 with the cursor message |
400 with the cursor message (unchanged, through the same rescue_from) |
Per-query circuit breaker (GlqlQueryLockedError) |
429 |
429 (unchanged) |
Side effect: a 500 no longer echoes the exception message to the client outside the test environment. Its body is {"message": "500 Internal Server Error"}, like every REST API 500, not the error key the other GLQL errors use. The docs now show it.
The two finder-limit errors are the only domain errors from GraphqlController::ERROR_STATUS_MAP that can escape GitlabSchema.execute on this path. The UI route returns 422 for them. This MR keeps the 400 the endpoint already returned, so they don't become another status change.
An unusable after cursor (for example one missing a value) also escapes as a plain exception, Gitlab::Pagination::Keyset::InvalidCursorError. It keeps its 400 through the same rescue_from. The UI route returns 500 for it.
API compatibility
The REST API style guide counts changing a status code other than 500 as a breaking change. This MR treats it as a bug fix: the GLQL API docs already document 400 for invalid input and 500 for server errors during query execution, so unexpected exceptions now match the documented contract. The 503 for timeouts is new.
Impact on the Data Analyst agent (run_glql_query)
No ai-assist change is required. The behaviour does change, because the Duo Workflow Service HTTP client retries 5xx.
Details
RunGLQLQuerypasses every non-2xx response throughDuoBaseTool._process_http_response, which raisesToolException("Request failed (GLQL API request): HTTP <code>: <body>"). No code branches on400. The only status-pinned test (HTTP 400for an invalid query) still holds, because invalid queries still return400.- Prompt:
<query_execution>says "If the query fails with a syntax error, fix the query and execute again". A timeout no longer looks like a syntax error.<performance>keys on the timeout, and theQuery timed outmessage is unchanged. - Retries:
ExecutorGitLabHttpClient._callretries5xx(and429) up to 4 attempts with 3s, 9s and 27s of backoff. Today a timeout400is never retried.- Timeout: the circuit breaker (
glqlrule: limit 1 per 15 minutes per query SHA,peekblocks whencount > limit) lets the slow query run twice, then answers429for the remaining attempts. The tool ends withHTTP 429: Query temporarily blocked due to repeated timeouts...after about 39s of extra waiting, and the same query stays blocked for the rest of the 15 minutes. - Unexpected exception: the query runs 4 times and produces 4 error-tracking events before the tool sees
HTTP 500.
- Timeout: the circuit breaker (
- Possible ai-assist follow-ups, not needed for this MR: stop retrying
503(or all5xx) for the GLQL tool, or have the tool turn a timeout into a "narrow the query" message.
Monitoring
gitlab_sli_rails_request_error_total counts every response with status >= 500 (Gitlab::Metrics.server_error?). Today GLQL API failures are 400s, so they never count as errors. After this MR, timeouts and exceptions count against the endpoint's error ratio. This is likely why Grafana shows no GLQL API errors while Kibana does (#595518 (closed)). This MR does not resolve that work item.
Known gaps (not changed here)
- The route still rescues
ArgumentErrorinto400, so anArgumentErrorraised during query execution still returns400. QueryServicemaps onlyActiveRecord::QueryAbortedto a timeout. A timeout from another backend (for example ClickHouse) arrives as a generic exception and becomes500. The UI route already behaves this way.
References
- Likely related: #595518 (closed)
- UI route behaviour:
app/controllers/glql/base_controller.rb - Agent tool:
ai/features/insights/analytics_agent/components/run_glql_query.pyandduo_workflow_service/gitlab/executor_http_client.pyingitlab-org/modelops/applied-ml/code-suggestions/ai-assist
Screenshots or screen recordings
Not applicable, API-only change.
How to set up and validate locally
-
Temporarily make execution fail: add
raise ActiveRecord::QueryCanceledas the first line ofexecute_graphqlinapp/services/analytics/glql/query_service.rb, thengdk restart rails-web. -
Run a query against a group you can read:
curl --request POST \ --header "PRIVATE-TOKEN: <your_access_token>" \ --header "Content-Type: application/json" \ --data '{"glql_yaml": "query: group = \"gitlab-org\" AND state = opened"}' \ --url "http://gdk.test:3000/api/v4/glql"It returns
503with{"error":"Query timed out"}. Onmasterit returns400. -
Replace the line with
raise 'boom', restart, and run the query again. It returns500, andlog/exceptions_json.loghas theRuntimeError. -
Remove the line and restart. A query with
fields: nonExistentFieldstill returns400.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.