Orbit web server ends running queries on shutdown instead of draining

Summary

When a web server pod receives SIGTERM, for example during a deploy, it stops at once and ends every query still running on it.

Expected behavior

On SIGTERM the web server stops taking new queries, sends GOAWAY, lets running queries finish for up to the 120 second maximum query time, and then exits. The pod's termination grace period is longer than that wait, so Kubernetes does not kill it mid-drain.

Actual behavior

The gRPC server stops as soon as the process gets SIGTERM. Open connections close without GOAWAY, and running query streams end with an error. Rails retries only when a stream fails before its first message, so queries that already started streaming fail for the user.

Found by reading the code. Not yet reproduced.

Steps to reproduce

  1. On staging, start a long-running query (30 seconds or more) through Rails.
  2. While it runs, restart the web server deployment (kubectl rollout restart).
  3. The query fails when its pod stops, instead of completing.

Environment

  • Orbit Remote: the hosted graph on GitLab.com (glab orbit remote, MCP, GitLab Duo, or the REST API).
  • Orbit Local: runs on your machine (the orbit binary or glab orbit local).

Orbit Remote

  • GitLab.com group or project path: any
  • Access method: any
  • Approximate time of the failure: every web server deploy or pod eviction

Logs

Expand logs

Not yet captured. Reproducing on staging will add them.

Workaround

None. Deploys at low traffic reduce the number of affected queries.

Additional context

Prerequisite for the Orbit Gateway (ADR 025, !3380). Behind the Gateway, Envoy moves new queries away from a stopping pod but cannot save running ones. Related: #971 (closed) (the same failure, caused by the connection age close).

Checklist

  • Reproduced on the latest available version.
  • Searched existing issues for duplicates.
  • Included minimal, reliable reproduction steps.
  • Attached relevant logs and environment details.
  • Removed sensitive information from logs/configs.
Agent context: extended analysis, hypothesis chains, file walkthroughs

Code path (knowledge-graph main, 2026-11-15):

  • crates/orbit-server/src/shutdown.rs: wait_for_signal catches SIGTERM or Ctrl+C and cancels a CancellationToken.
  • crates/orbit-server/src/main.rs:327: run_webserver runs tokio::select! over grpc_server.run(grpc_listener) and shutdown.cancelled(). When the token is cancelled, the select drops the gRPC server future, which closes every connection at once.
  • crates/orbit-server/src/grpc/server.rs: the server is started with serve_with_incoming(incoming). Tonic's serve_with_incoming_shutdown(incoming, signal) is the variant that stops accepting, sends GOAWAY, and waits for in-flight streams.
  • orbit-helm-charts sets no terminationGracePeriodSeconds or preStop for the web server, so the Kubernetes default of 30 seconds applies.

Proposed fix:

  1. Pass the shutdown token to the gRPC server and use serve_with_incoming_shutdown, so SIGTERM starts a graceful drain instead of dropping the server.
  2. Bound the drain with a configurable timeout above the 120 second maximum query time (query timeout plus margin), after which remaining streams are cancelled.
  3. Keep billing, analytics, and quota trackers flushing after the drain, as run_webserver does today.
  4. In orbit-helm-charts, set the web server terminationGracePeriodSeconds above the drain timeout. Optionally add a short preStop sleep so endpoint removal propagates before GOAWAY.
  5. Test: an integration test that starts a slow stream, sends SIGTERM, and asserts the stream completes and new connections are refused.

Relation to max_connection_age: separate mechanism. The connection age is a per-connection timer while the pod keeps running; this is the whole-pod shutdown path. ADR 025 turns the connection age off behind the Gateway, which makes this drain the only path that must protect running queries.

Duplicates searched: #971 (closed) (closed, connection age close in the Workhorse path), #511 (closed) (closed, tonic panic on connection age), #985 (closed) (closed, blue-green rollout). None covers shutdown on SIGTERM.