fix(workhorse): return 402 instead of 502 when GKG quota is exhausted (!236988) · Merge requests · GitLab.org / GitLab

Summary

When the GKG gRPC stream returns codes.ResourceExhausted (i.e. the user's GitLab credits are exhausted), Workhorse was falling through to the generic handleRecvError 502 Bad Gateway path. This gave callers no actionable information.

This MR adds explicit handling for ResourceExhausted in orbit/sendquery.go:

REST callers: HTTP 402 Payment Required with a structured JSON body {"code":"quota_exhausted","message":"..."}.
MCP callers: HTTP 200 OK with a JSON-RPC 2.0 error envelope (isError: true) — per the MCP spec, tool-level errors are signalled in the result body, not via HTTP status codes.
~~If GKG returns an empty message, Workhorse falls back to "GitLab credits exhausted".~~

Testing locally

Prerequisites: Local GDK running Rails and gitlab-workhorse, plus a local GKG (Knowledge Graph) server checked out from gitlab-org/orbit/knowledge-graph.
Rebuild the Workhorse binary. Restarting the process alone does not pick up Go source changes:
```
cd workhorse && make gitlab-workhorse
gdk restart gitlab-workhorse
```

Start the CustomersDot stub that always returns HTTP 402. Save this as cdot_402_stub.py:

import http.server

class Handler(http.server.BaseHTTPRequestHandler):
    def do_HEAD(self):
        self.send_response(402)
        self.send_header("Content-Length", "0")
        self.end_headers()

    def do_GET(self):
        self.do_HEAD()

if __name__ == "__main__":
    server = http.server.HTTPServer(("127.0.0.1", 5099), Handler)
    print("cdot-402-stub listening on 127.0.0.1:5099, always returns 402")
    server.serve_forever()

Then run: python3 cdot_402_stub.py

Start GKG from the knowledge-graph repo root with billing quota enabled. Read your GDK's .gitlab_knowledge_graph_secret file and use its base64 contents for both signing and verifying keys (Rails and GKG must use the same secret):

GKG_BILLING__QUOTA__ENABLED=true \
GKG_BILLING__QUOTA__CUSTOMERS_DOT_URL=http://127.0.0.1:5099 \
GKG_BILLING__QUOTA__API_USER=stub@example.com \
GKG_BILLING__QUOTA__API_TOKEN=stub-token \
GKG_BILLING__QUOTA__REQUEST_TIMEOUT_MS=1000 \
GKG_GITLAB__BASE_URL=http://localhost:3000 \
GKG_GITLAB__JWT__SIGNING_KEY=$(cat /path/to/gdk/gitlab/.gitlab_knowledge_graph_secret) \
GKG_GITLAB__JWT__VERIFYING_KEY=$(cat /path/to/gdk/gitlab/.gitlab_knowledge_graph_secret) \
mise run server:start

Generate a personal access token with api and mcp scopes:

# bin/rails runner
user = User.find_by(username: '<your-username>')
pat = user.personal_access_tokens.create!(name: 'orbit-quota-test', scopes: ['api', 'mcp'], expires_at: 1.day.from_now)
puts pat.token

Test the REST path. Expect HTTP 402 with the quota_exhausted error code:

curl -s -X POST http://localhost:3000/api/v4/orbit/query \
  -H "Authorization: Bearer <PAT>" \
  -H "Content-Type: application/json" \
  -d '{"query":{"query_type":"neighbors","nodes":[{"id":"me","entity":"User","node_ids":[<user_id>]}],"neighbors":{"direction":"both"},"limit":5},"response_format":"raw"}' \
  -w "\nHTTP_STATUS:%{http_code}\n"

Expected: {"code":"quota_exhausted","message":"GitLab credits exhausted"} with HTTP_STATUS:402.

Test the MCP path. Expect HTTP 200 with a JSON-RPC envelope containing isError:true:

curl -s -X POST http://localhost:3000/api/v4/orbit/mcp \
  -H "Authorization: Bearer <PAT>" \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"invoke_command","arguments":{"command_name":"query_graph","parameters":{"query":{"query_type":"neighbors","nodes":[{"id":"me","entity":"User","node_ids":[<user_id>]}],"neighbors":{"direction":"both"},"limit":5},"format":"raw"}}}}' \
  -w "\nHTTP_STATUS:%{http_code}\n"

Expected: {"jsonrpc":"2.0","result":{"content":[{"type":"text","text":"GitLab credits exhausted"}],"isError":true},"id":1} with HTTP_STATUS:200.

Edited Aug 18, 2026 by Sharmad Nachnolkar

fix(workhorse): return 402 instead of 502 when GKG quota is exhausted

Summary

Testing locally

Merge request reports