Return 503 and 500 from the GLQL API for server failures

What does this MR do and why?

POST /api/v4/glql answers every execution failure with 400 Bad Request, including statement timeouts and unexpected exceptions. Clients, including the Data Analyst agent, are told their query was invalid when the server failed. The failures also never reach error tracking or the Rails request error SLI, and the Kibana "GLQL query validity rate" counts slow queries and bugs as the agent writing invalid GLQL.

This MR handles these cases the way the UI's GraphQL route (Glql::BaseController) already does:

Failure Before After
Statement timeout (ActiveRecord::QueryAborted) 400, Query timed out 503, Query timed out
Unexpected exception during execution 400 with the exception message re-raised, so the API's normal exception handling applies: 500 plus error tracking, unless Grape has a more specific handler (for example ActiveRecord::RecordNotFound becomes 404)
YAML parse, compile, GraphQL validation or field, and transform errors 400 400 (unchanged)
Group filter over a finder limit (TooManyGroupMembersError, TooManyAssignedIssuesError) 400 with the limit message 400 with the limit message (unchanged, through a rescue_from in API::Glql)
Unusable after cursor (Gitlab::Pagination::Keyset::InvalidCursorError) 400 with the cursor message 400 with the cursor message (unchanged, through the same rescue_from)
Per-query circuit breaker (GlqlQueryLockedError) 429 429 (unchanged)

Side effect: a 500 no longer echoes the exception message to the client outside the test environment. Its body is {"message": "500 Internal Server Error"}, like every REST API 500, not the error key the other GLQL errors use. The docs now show it.

The two finder-limit errors are the only domain errors from GraphqlController::ERROR_STATUS_MAP that can escape GitlabSchema.execute on this path. The UI route returns 422 for them. This MR keeps the 400 the endpoint already returned, so they don't become another status change.

An unusable after cursor (for example one missing a value) also escapes as a plain exception, Gitlab::Pagination::Keyset::InvalidCursorError. It keeps its 400 through the same rescue_from. The UI route returns 500 for it.

API compatibility

The REST API style guide counts changing a status code other than 500 as a breaking change. This MR treats it as a bug fix: the GLQL API docs already document 400 for invalid input and 500 for server errors during query execution, so unexpected exceptions now match the documented contract. The 503 for timeouts is new.

Impact on the Data Analyst agent (run_glql_query)

No ai-assist change is required. The behaviour does change, because the Duo Workflow Service HTTP client retries 5xx.

Details
  • RunGLQLQuery passes every non-2xx response through DuoBaseTool._process_http_response, which raises ToolException("Request failed (GLQL API request): HTTP <code>: <body>"). No code branches on 400. The only status-pinned test (HTTP 400 for an invalid query) still holds, because invalid queries still return 400.
  • Prompt: <query_execution> says "If the query fails with a syntax error, fix the query and execute again". A timeout no longer looks like a syntax error. <performance> keys on the timeout, and the Query timed out message is unchanged.
  • Retries: ExecutorGitLabHttpClient._call retries 5xx (and 429) up to 4 attempts with 3s, 9s and 27s of backoff. Today a timeout 400 is never retried.
    • Timeout: the circuit breaker (glql rule: limit 1 per 15 minutes per query SHA, peek blocks when count > limit) lets the slow query run twice, then answers 429 for the remaining attempts. The tool ends with HTTP 429: Query temporarily blocked due to repeated timeouts... after about 39s of extra waiting, and the same query stays blocked for the rest of the 15 minutes.
    • Unexpected exception: the query runs 4 times and produces 4 error-tracking events before the tool sees HTTP 500.
  • Possible ai-assist follow-ups, not needed for this MR: stop retrying 503 (or all 5xx) for the GLQL tool, or have the tool turn a timeout into a "narrow the query" message.

Monitoring

gitlab_sli_rails_request_error_total counts every response with status >= 500 (Gitlab::Metrics.server_error?). Today GLQL API failures are 400s, so they never count as errors. After this MR, timeouts and exceptions count against the endpoint's error ratio. This is likely why Grafana shows no GLQL API errors while Kibana does (#595518 (closed)). This MR does not resolve that work item.

Known gaps (not changed here)

  • The route still rescues ArgumentError into 400, so an ArgumentError raised during query execution still returns 400.
  • QueryService maps only ActiveRecord::QueryAborted to a timeout. A timeout from another backend (for example ClickHouse) arrives as a generic exception and becomes 500. The UI route already behaves this way.

References

  • Likely related: #595518 (closed)
  • UI route behaviour: app/controllers/glql/base_controller.rb
  • Agent tool: ai/features/insights/analytics_agent/components/run_glql_query.py and duo_workflow_service/gitlab/executor_http_client.py in gitlab-org/modelops/applied-ml/code-suggestions/ai-assist

Screenshots or screen recordings

Not applicable, API-only change.

How to set up and validate locally

  1. Temporarily make execution fail: add raise ActiveRecord::QueryCanceled as the first line of execute_graphql in app/services/analytics/glql/query_service.rb, then gdk restart rails-web.

  2. Run a query against a group you can read:

    curl --request POST \
      --header "PRIVATE-TOKEN: <your_access_token>" \
      --header "Content-Type: application/json" \
      --data '{"glql_yaml": "query: group = \"gitlab-org\" AND state = opened"}' \
      --url "http://gdk.test:3000/api/v4/glql"

    It returns 503 with {"error":"Query timed out"}. On master it returns 400.

  3. Replace the line with raise 'boom', restart, and run the query again. It returns 500, and log/exceptions_json.log has the RuntimeError.

  4. Remove the line and restart. A query with fields: nonExistentField still returns 400.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Daniele Rossetti

Merge request reports

Loading
Loading