Orbit query 502s: max_connection_age force-close kills in-flight Workhorse streams
Summary
POST /api/v4/orbit/query returns intermittent 502s because the webserver force-closes every gRPC connection at a fixed age (300s plus a 30s grace) while Workhorse multiplexes all Orbit queries over one shared connection with no retry. Whatever is in flight at the 330s boundary dies.
Expected behavior
Connection recycling never fails a request. In-flight streams drain before the connection closes, and new streams open on a fresh connection.
Actual behavior
At connection age 330s, every in-flight ExecuteQuery stream is killed. Workhorse logs
{"error":"orbit.SendQuery: stream recv: rpc error: code = Unavailable desc = error reading from server: EOF","level":"error","uri":"/api/v4/orbit/query","time":"2026-07-03T20:13:06Z"}and returns 502. GKG records the same instant as an authorization failure, which it is not:
{"level":"ERROR","message":"Pipeline error","code":"authorization_error","failure_reason":"authorization","error":"Authorization failed: ReceiveFailed(Status { code: Unknown, message: \"h2 protocol error: error reading a body from connection\", source: ... BrokenPipe ... })","target":"gkg_server::pipeline::helpers"}Steps to reproduce
- Send steady traffic to
POST /api/v4/orbit/query. Any e2e run does. - Note the timestamp of the first request. That request creates Workhorse's cached connection.
- Watch for 502s at that timestamp plus 330 seconds.
Both e2e occurrences to date match to the second:
| e2e job | first orbit query | 502s fire | delta |
|---|---|---|---|
| 15174409921 (MR !1992 (merged) pipeline) | 20:07:36 | 20:13:06 | 330s |
| 15134773024 (MR !1968 (merged) pipeline) | 18:59:29 | 19:04:59 | 330s |
Environment
- e2e harness (GKE, chart-deployed webserver). Production runs the same defaults, so this is not an e2e-only problem.
- GKG main @ a0527689.
Logs
Expand logs (job 15174409921, 2026-07-03T20:13:06Z)
Workhorse, both in-flight requests on the shared connection:
{"correlation_id":"01KWMST3E1J0E6TWJXCBTDFBTY","error":"orbit.SendQuery: stream recv: rpc error: code = Unavailable desc = error reading from server: EOF","level":"error","method":"POST","time":"2026-07-03T20:13:06Z","uri":"/api/v4/orbit/query"}
{"correlation_id":"01KWMST3DMC4ZDHA5Z8481Q4KG","error":"orbit.SendQuery: stream recv: rpc error: code = Unavailable desc = error reading from server: EOF","level":"error","method":"POST","time":"2026-07-03T20:13:06Z","uri":"/api/v4/orbit/query"}GKG, first victim (killed mid redaction exchange):
{"timestamp":"2026-07-03T20:13:06.583438Z","level":"ERROR","message":"Failed to receive redaction response","error":"code: 'Unknown error', message: \"h2 protocol error: error reading a body from connection\", source: ... BrokenPipe ...","target":"gkg_server::redaction::stream","spans":[{"correlation_id":"01KWMST3E1J0E6TWJXCBTDFBTY","name":"grpc.request"}]}The second victim (01KWMST3DM...) finished server-side, GKG logged "No redaction required, returning result directly" and "Sending final query result", and Workhorse still got EOF reading the reply. Both ends saw a broken read at the same instant, so neither closed the connection gracefully. Rails logged zero 5xx for the whole run; 2 of 556 orbit POSTs failed, both inside this one second; the next request 0.4s later succeeded on a fresh connection. In the MR !1968 (merged) job the same teardown also killed a third stream mid redaction send (orbit.SendQuery: send redaction response: EOF).
Workaround
Retry the request. MR !1997 (merged) adds a single retry on 502 to the e2e query keywords, which removes the flake from CI but not from production callers.
Additional context
Rails was the first suspect and the first dead end: its gRPC error mapping tops out at 503/504 and its logs showed nothing but 200s. The 502 comes from Workhorse itself, which proxies /api/v4/orbit/query straight to GKG over gRPC after Rails authorizes the request. From there the trail is three code reads long.
The webserver recycles connections on purpose (config/default.yaml#L149-150, applied in grpc/server.rs#L103-110):
max_connection_age_secs: 300 # 5 min, for L4 ILB rebalancing
max_connection_age_grace_secs: 30 # graceful drain window; paired with max_connection_age_secs
# to avoid tonic 0.14.5 panic (hyperium/tonic#2522)Workhorse holds one cached connection per GKG address and, by its own comment, counts on the server to make recycling safe (workhorse/internal/orbit/client.go#L55-59):
// getOrCreateConnection returns a cached gRPC connection or creates a new one.
// Connections in Shutdown or TransientFailure state are replaced.
// Connection rebalancing across GKG pods is handled server-side via
// MAX_CONNECTION_AGE in tonic (see .../knowledge-graph/-/work_items/330).
func getOrCreateConnection(server GkgServer) (*grpc.ClientConn, error) {And when the recycled connection takes a stream down with it, Workhorse has no second attempt; any receive error that is not a timeout becomes a 502 (workhorse/internal/orbit/sendquery.go#L172-183):
func handleRecvError(ctx context.Context, w http.ResponseWriter, r *http.Request, err error) {
if err == io.EOF { ... http.StatusBadGateway ... }
if isContextDone(ctx, err) { ... http.StatusGatewayTimeout ... }
log.WithRequest(r).WithError(fmt.Errorf("orbit.SendQuery: stream recv: %v", err)).Error()
fail.Request(w, r, fmt.Errorf("orbit.SendQuery: stream error"), fail.WithStatus(http.StatusBadGateway))
}Three fixes fall out of this, in value order:
- Workhorse: retry the whole exchange once on
Unavailable. It is safe because nothing has been written to the HTTP client before the first result and both the query and the redaction callback are idempotent reads. A gRPC service-config retryPolicy is not enough, since bidi streams stop being retryable after the first message. - GKG: stop classifying transport failures as authorization failures. authorization.rs#L56-59 collapses every redaction-exchange error into
PipelineError::Authorization, even thoughRedactionExchangeError::into_status()already mapsReceiveFailed/StreamClosedcorrectly. Today an infra drop pollutes thefailure_reason="authorization"metric and log stream. - Find out why streams that opened 29s into the grace window were still routed onto the dying connection. Either the GOAWAY goes out late (tonic) or grpc-go does not act on it. A local repro against tonic 0.14 would settle which side to fix.
Bookkeeping note for whoever picks this up: the client.go comment and ADR 008 both cite #330 (closed) as the source of the max-age design, but #330 (closed) is HTTP/2 concurrency tuning and never mentions connection age. The setting actually dates to #511 (closed).
Related: #570 is a different 502 from the redaction callback, #767 (closed) would give this failure a structured error shape instead of bare Bad Gateway, #511 (closed) is where the grace pairing came from, #349 (closed) introduced the shared no-retry client, and #967 (closed) is the e2e work that kept tripping over this.
Checklist
- Used latest available version of gkg (main @ a0527689)
- Searched existing issues for duplicates (none track this; closest are #570 and #767 (closed))
- Included minimal, reliable reproduction steps
- Attached relevant logs and environment details
- Removed sensitive information from logs/configs
Agent context — extended analysis, hypothesis chains, file walkthroughs
Full request dataflow: robot/API client POSTs /api/v4/orbit/query → Workhorse forwards to Rails → Rails authorizes (ee/lib/api/orbit/data.rb), mints a short-lived JWT, answers 200 with a Gitlab-Workhorse-Send-Data: orbit-query:<params> header → Workhorse hijacks the response and opens bidirectional ExecuteQuery on the cached shared *grpc.ClientConn (workhorse/internal/orbit/client.go#L47-93) → GKG pipeline runs Security → PathResolution → Compilation → ClickHouse → Extraction → Authorization → Redaction → Hydration → Output (crates/gkg-server/src/pipeline/service.rs) → when redaction is needed GKG sends RedactionRequired back down the stream and blocks on the reply (crates/gkg-server/src/redaction/stream.rs#L110-121); Workhorse answers it by POSTing Rails POST /api/v4/internal/orbit/redaction and relaying the result → GKG sends the final result.
Evidence that the teardown is connection-level, not stream- or app-level: (a) both victims failed inside the same second with different failure points (one mid redaction exchange, one after the server sent its final result); (b) Rails api_json logs show zero 5xx across the run and 200s for the same correlation ids (the preflight); (c) gkg-webserver had zero restarts and no other errors; (d) ClickHouse state was clean; (e) the very next requests succeeded 0.4s later. The 330s deltas were measured from the Workhorse access log (first /api/v4/orbit/query entry) to the 502 access-log entries, on both job 15174409921 (e2e diagnostics artifact) and job 15134773024.
Keepalive settings for completeness: http2_keepalive_interval 20s, http2_keepalive_timeout 20s, tcp_keepalive 60s (config/default.yaml#L143-145). A missed keepalive would produce the same signature but not the exact 330s periodicity observed twice.
Chart note: the gkg Helm chart (0.24.0, and main as of this writing) renders a fixed webserver config with no grpc: section and no env passthrough, so chart deploys take the Rust defaults from crates/gkg-server-config/src/grpc.rs (300/30). There is currently no values-level knob to change connection age in e2e or production without a chart change.
Investigation session: 71f7eb2b-4dab-4cc5-adae-7859fede0c8e (root cause, dataflow, timing proof).