## Symptom When the RabbitMQ container restarts (crash, OOM kill, compose-driven replacement, or planned upgrade), every `canopy-*` service that holds an AMQP `Connection` / `Channel` via `canopy-mq` enters a permanently-broken state. Subsequent publishes fail with `AMQP error: invalid channel state: Error (basic.publish)` and the service never reconnects until it is itself restarted. Observable consequences: - Cross-service event flows (SNAP → TSNAP, TANF → TMA, TANF → Medicaid Express Lane) silently stop propagating. - Integration tests driven by `poll_until` time out after 15s (`TSNAP certification for household`, `ELE evaluation for household`, etc.) because the subscriber side never sees the event. - In production this would manifest as **lost events**: services would keep accepting HTTP requests and writing DB state, but the event-bus notifications meant to trigger downstream workflows (notices, enrollment, auditing, breach detection) would never fire. ## Evidence from 2026-04-20 xtask refactor session (MR !79) canopy-tanf log tail after RabbitMQ was replaced by `docker compose run --build canopy-e2e`: ``` ERROR Connection closed channel=0 method=Close { reply_code: 320, reply_text: "CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'" } ERROR Connection error: protocol error: AMQP hard error: CONNECTION-FORCED WARN failed to publish tanf.determined event: AMQP error: invalid channel state: Error (basic.publish) WARN failed to publish tanf.application_approved: AMQP error: invalid channel state WARN failed to publish tanf.case_closed event: AMQP error: invalid channel state ``` lapin correctly detected the connection drop, but `EventPublisher::publish()` at `crates/canopy-mq/src/publisher.rs` uses a static `Channel` handle that is poisoned after the disconnect. No reconnect attempt is made. MR !79 sidestepped this by preventing unnecessary RabbitMQ container recreation (split `canopy-e2e` build from run, stopped `run --build` from rebuilding the depends_on closure). But the underlying fragility remains: if anything else restarts RabbitMQ (crash, memory pressure, planned maintenance, an ops engineer running `docker compose restart rabbitmq`), the same cascade occurs. ## Current architecture - `crates/canopy-mq/src/lib.rs:32` — `connect(url) -> Result<Connection, lapin::Error>` — one-shot, no retry - `crates/canopy-mq/src/publisher.rs` — `EventPublisher` wraps a `Channel`, constructed once at service boot, held for the service lifetime - `crates/canopy-mq/src/subscriber.rs` — subscribers create a `Channel` and spawn a consume loop; when the loop's delivery-stream errors, the task logs `consumer delivery error` and exits (never retries) Every `canopy-*` service's `main.rs` calls `canopy_mq::connect` once, then shares the resulting `Connection` / `Channel` through its `AppState`. There's no reconnect supervisor, no channel-level error recovery, no publisher retry queue. ## Acceptance criteria 1. **Reconnect on connection drop.** A supervisor task monitors the lapin `Connection` status; when it detects close / IO error, it transparently reconnects with exponential backoff (documented bounds, e.g. 100ms → 30s max, indefinite retries). 2. **Channel re-creation.** After reconnect, the `EventPublisher` and every active `Subscriber` re-creates its `Channel`, re-declares any bindings, and resumes operation. Publishers hold a reference that points at the *current* channel, not a copy of the dead one. 3. **Publish resilience.** `EventPublisher::publish` either: - queues the message in-memory and flushes after reconnect (preferred — no lost events), or - returns a retryable error that callers can surface or enqueue themselves (minimum — callers can decide). The current behavior (silent `WARN` with dropped event) is not acceptable for compliance-relevant flows like `determination.completed` or FTI audit events. 4. **Subscriber resilience.** `Subscriber` consume loops reattach to the new channel after reconnect instead of exiting. 5. **Observability.** Log the reconnect attempts and successes at INFO, failures at ERROR with structured `attempt` / `backoff_ms` fields. 6. **Integration test.** A test in `canopy-security` (or a dedicated `canopy-mq` integration test) that: starts the devstack, publishes an event, `docker compose restart rabbitmq`, publishes another event, asserts both events were persisted by the subscriber — proves the full publish → broker restart → publish → delivery chain survives. ## Out of scope - Publisher backpressure / at-least-once semantics beyond the in-memory queue in (3). Proper persistent outbox is a separate ADR-worthy design. - Dead-letter handling for messages the subscriber can't process. - RabbitMQ HA cluster / mirrored queues. ## Related - MR !79 (refactor/xtask-unsafe-env-removal) — sidestepped this bug by preventing unnecessary RabbitMQ recreation in the `xtask e2e` path - Issue #310 — postgres anonymous volumes (same class of "service restart breaks downstream state"; fixed via named volumes) - RCA trail: the TSNAP/TMA/ELE E2E tests have been flaky on pre-push hooks since at least 2026-04-18; this bug is likely the root cause of all of them ## Labels `workflow::needs-spec` — design for the reconnect supervisor + publisher-queue semantics should be sketched in a plan (or ADR if we land the at-least-once outbox) before implementation begins.