A message backlog OOM-kills the worker's PostgreSQL: consumption is unbounded (prefetch 0) and the pool allows 100 connections

Symptom

On staging, right after the upgrade to 26.10.0-rc2, the Kubernetes worker's PostgreSQL was OOMKilled three times in a row: ryax-worker-k8s-postgresql-0, last state OOMKilled at 2026-10-06 15:08 UTC. The worker logged asyncpg connection timeouts in its handlers (on_stop_execution → TimeoutError).

The trigger was a backlog. While rc1's broker was blocked (ryax-engine!100 (merged)), 58 messages piled up in WorkerQ.<site>. When the broker came back, the worker took all 58 at once (58 unacknowledged deliveries) and handled them concurrently, each with its own database connection. PostgreSQL ran out of memory, restarted, and the queue only drained after the third restart.

Cause, confirmed in the code (ryax-core @ bb1915b9)

  • Message consumption is deliberately unbounded. ryax/common/messaging/consumer.py:110: 
    # Do not bound message consumption
await self.channel.set_qos(prefetch_count=0)
    RabbitMQ then pushes every queued message to the consumer at once.
  • The worker's database pool allows up to 100 connections. ryax/worker_k8s/infrastructure/database/async_engine.py:43-44: pool_size=10, max_overflow=90. That is PostgreSQL's default max_connections.
  • The worker's PostgreSQL is tiny. It is the Bitnami postgresql subchart of the worker chart (bitnamilegacy/postgresql:17.6.0), with limits of 150m CPU and 192Mi. Each backend costs a few MB, so a burst of tens of connections exceeds it.

In normal operation the queue is near empty and none of this shows. Any backlog does trigger it: a broker or worker outage, a burst of executions, or a large workflow stopped at once.

Options (to decide)

  1. Bound consumption. A finite prefetch_count on the consumer (per service, configurable), so a backlog is processed at a bounded concurrency. This is the actual fix.
  2. Bound the pool. A max_overflow matching what the database can serve.
  3. Size the worker database. Raise the worker chart's PostgreSQL limits (postgresql.primary.resources), or move it off the Bitnami chart as part of #1320, which would set the resources anyway.

Related: #1320 (the worker charts still use the Bitnami postgresql chart), #1475 (closed) (26.10.0 release; seen during the staging upgrade).