perf(api): WebSocket reconnect-storm scaling — shard channel-layer Redis, coalesce presence, batch replay

Problem

A deploy, ingress restart, or network blip mass-reconnects a large project's live collaborators, and three coupled behaviors compound through a single, unsharded channel-layer Redis:

  1. Single Redis channel layer is the global fan-out bottleneck + SPOF. settings/base.py:286 sets hosts: [f"{REDIS_URL}/1"] — a one-element list, so channels_redis has no sharding, and settings/prod.py does not override it. Every group_send for every project funnels through one instance, which also shares the box with the cache (/2) and Celery broker (/0).
  2. Presence broadcast is O(M²). Every websocket_connect calls _presence_join() which broadcasts presence_join to the entire project group (sync/consumers.py:308); every disconnect broadcasts presence_leave (:324). No coalescing, no rate-limit. M members reconnecting at once = M broadcasts × ~M recipients + M role-lookup queries.
  3. Replay thundering herd. Each reconnecting client with ?since= streams up to _REPLAY_CAP = 1000 BoardEvent rows one await self.send_json(...) at a time (sync/consumers.py:268). N clients × up to 1000 frames simultaneously.

Nothing on the server coalesces presence, and nothing on the client staggers reconnects.

Scale at which it bites

A large program's project with tens-to-low-hundreds of concurrent collaborators, during a deploy or connectivity blip. Beta-scale (a handful of users) does not hit this.

Why 0.5

The 0.5 installable PWA (#1393, #1427) makes reconnect-with-queued-writes a first-class flow, and push notifications (#2132) add another connection surface — 0.5 is the forcing function.

Proposed fix (coupled — warrants an ADR)

  • Shard or isolate the channel-layer Redis: multi-node hosts list, or a dedicated Valkey separate from cache/broker, with a documented throughput ceiling.
  • Coalesce presence: replace per-connect/disconnect broadcasts with a periodic diffed presence snapshot on a timer.
  • Batch replay into a single framed message (array of events) instead of N send_json calls; consider a smaller cap before falling back to resync_required.
  • Add jittered client reconnect backoff so a mass reconnect spreads over seconds.

Files

  • packages/api/src/trueppm_api/apps/sync/consumers.py (presence :295-333, replay :205-277)
  • packages/api/src/trueppm_api/settings/base.py:286

Related

  • #1584 (closed) (ProjectConsumer per-presence-call Redis pool — the per-frame leak was addressed under #1530 (closed); this issue is the broader topology/fan-out concern)
  • #1427 (0.5 offline shell / reconnect banner)

Verified-healthy and out of scope: bulk ops already emit one coalesced broadcast (not per-row); broadcasts are wrapped in transaction.on_commit; async consumers use database_sync_to_async throughout.