Orbit web server ends running queries on shutdown instead of draining
Summary
When a web server pod receives SIGTERM, for example during a deploy, it stops at once and ends every query still running on it.
Expected behavior
On SIGTERM the web server stops taking new queries, sends GOAWAY, lets running queries finish for up to the 120 second maximum query time, and then exits. The pod's termination grace period is longer than that wait, so Kubernetes does not kill it mid-drain.
Actual behavior
The gRPC server stops as soon as the process gets SIGTERM. Open connections close without GOAWAY, and running query streams end with an error. Rails retries only when a stream fails before its first message, so queries that already started streaming fail for the user.
Found by reading the code. Not yet reproduced.
Steps to reproduce
- On staging, start a long-running query (30 seconds or more) through Rails.
- While it runs, restart the web server deployment (
kubectl rollout restart). - The query fails when its pod stops, instead of completing.
Environment
- Orbit Remote: the hosted graph on GitLab.com (
glab orbit remote, MCP, GitLab Duo, or the REST API). - Orbit Local: runs on your machine (the
orbitbinary orglab orbit local).
Orbit Remote
- GitLab.com group or project path: any
- Access method: any
- Approximate time of the failure: every web server deploy or pod eviction
Logs
Expand logs
Not yet captured. Reproducing on staging will add them.
Workaround
None. Deploys at low traffic reduce the number of affected queries.
Additional context
Prerequisite for the Orbit Gateway (ADR 025, !3380). Behind the Gateway, Envoy moves new queries away from a stopping pod but cannot save running ones. Related: #971 (closed) (the same failure, caused by the connection age close).
Checklist
- Reproduced on the latest available version.
- Searched existing issues for duplicates.
- Included minimal, reliable reproduction steps.
- Attached relevant logs and environment details.
- Removed sensitive information from logs/configs.
Agent context: extended analysis, hypothesis chains, file walkthroughs
Code path (knowledge-graph main, 2026-11-15):
crates/orbit-server/src/shutdown.rs:wait_for_signalcatches SIGTERM or Ctrl+C and cancels aCancellationToken.crates/orbit-server/src/main.rs:327:run_webserverrunstokio::select!overgrpc_server.run(grpc_listener)andshutdown.cancelled(). When the token is cancelled, the select drops the gRPC server future, which closes every connection at once.crates/orbit-server/src/grpc/server.rs: the server is started withserve_with_incoming(incoming). Tonic'sserve_with_incoming_shutdown(incoming, signal)is the variant that stops accepting, sends GOAWAY, and waits for in-flight streams.orbit-helm-chartssets noterminationGracePeriodSecondsorpreStopfor the web server, so the Kubernetes default of 30 seconds applies.
Proposed fix:
- Pass the shutdown token to the gRPC server and use
serve_with_incoming_shutdown, so SIGTERM starts a graceful drain instead of dropping the server. - Bound the drain with a configurable timeout above the 120 second maximum query time (
querytimeout plus margin), after which remaining streams are cancelled. - Keep billing, analytics, and quota trackers flushing after the drain, as
run_webserverdoes today. - In
orbit-helm-charts, set the web serverterminationGracePeriodSecondsabove the drain timeout. Optionally add a shortpreStopsleep so endpoint removal propagates before GOAWAY. - Test: an integration test that starts a slow stream, sends SIGTERM, and asserts the stream completes and new connections are refused.
Relation to max_connection_age: separate mechanism. The connection age is a per-connection timer while the pod keeps running; this is the whole-pod shutdown path. ADR 025 turns the connection age off behind the Gateway, which makes this drain the only path that must protect running queries.
Duplicates searched: #971 (closed) (closed, connection age close in the Workhorse path), #511 (closed) (closed, tonic panic on connection age), #985 (closed) (closed, blue-green rollout). None covers shutdown on SIGTERM.