gRPC server panics on max_connection_age expiry due to tonic 0.14.5 bug (hyperium/tonic#2522)
Every time a gRPC connection hits max_connection_age, the server panics with:
thread 'tokio-rt-worker' panicked at .../tonic-0.14.5/src/transport/server/mod.rs:891:20:
`async fn` resumed after completionThe process keeps serving traffic because tokio catches the panic, but the graceful-shutdown that backs our L4 ILB rebalancing never actually completes, and the panic noise buries real ERROR signal.
Upstream bug: hyperium/tonic#2522. Open fix hyperium/tonic!2588 is unmerged and unreleased as of 2026-04-23.
Why it fires for us
We set max_connection_age without max_connection_age_grace, which is the exact repro condition in the upstream issue.
- crates/gkg-server/src/grpc/server.rs#L70-78 calls
.max_connection_age(...). - crates/gkg-server-config/src/grpc.rs has no grace field, default age 300s.
Panic timing correlates directly with connection age reaching the configured 300s, not with any request shape. The 1:1 ratio between panics and requests we observe is incidental: our request cadence is roughly one per connection lifetime, so each connection panics once when its age timer fires.
Bug mechanism in tonic 0.14.5
The select loop in tonic 0.14.5 mod.rs:938-961:
let mut connection_timeout = pin!(connection_timeout_future(
max_connection_age,
max_connection_age_grace,
));
loop {
tokio::select! {
rv = &mut conn => { ...; break; },
timeout_action = &mut connection_timeout => {
match timeout_action {
TimeoutAction::GracefulShutdown => {
conn.as_mut().graceful_shutdown();
// no break here
},
TimeoutAction::ForcefulShutdown => { break; }
}
},
_ = &mut sig => { conn.as_mut().graceful_shutdown(); },
}
}With grace unset, connection_timeout_future returns TimeoutAction::GracefulShutdown. The match arm does not break. The loop iterates and select! polls the already-completed future again. Rust's async lowering panics on poll-after-ready. The sibling sig future is wrapped in Fuse at mod.rs:927-929 to avoid exactly this; PR #2588 adds the same wrapper around connection_timeout.
Fix
Preferred: add a grace-period field and wire it into the builder.
- Add
max_connection_age_grace_secs: Option<u64>toGrpcConfig. - In the server builder, chain
.max_connection_age_grace(Duration::from_secs(v))when the option is set. - Default the new field to something small but non-zero (~30s) so the graceful drain window is respected and the bug stops firing.
With both values set, connection_timeout_future returns ForcefulShutdown, which hits the break arm. No re-poll, no panic.
Fallback if the above is not viable:
- Pin
tonic = "=0.13.1". Clean, but gives up everything added in 0.14.x. - Wait for hyperium/tonic!2588. Open, not near-term.
Acceptance
- No
`async fn` resumed after completionpanics under normal operation. -
GrpcConfigsupportsmax_connection_age_grace_secswith a non-zero default. - Existing 300s
max_connection_age_secsdefault preserved so L4 ILB rebalancing still fires. - Test that exercises connection-age expiry and asserts no panic.
Parent epic: gitlab-org&20992.