2026-03-04: Duo agentic chat losing messages with some models

Duo agentic chat losing messages with some models (Severity 2)

Problem: Agentic chat in the Duo Agentic Platform intermittently failed to return responses after users sent messages. Requests appeared to start but then stalled, with no indication in the UI or logs.

Impact: Multiple users in both the Duo Agentic Platform web UI and CLI experienced missing responses from agentic chat, with sessions stalling and no visible errors. Two flagship customers, including T-Mobile, were blocked from giving demos. Following the fixes, chat reliability was restored and error rates returned to normal.

Causes: Unhealthy workflow service pods could not verify authentication tokens if they failed to fetch public keys from gitlab.com at startup. The initial fix removed in-memory caching to avoid persistent failures, but this added latency. Later, caching was reintroduced with a change: if fetching public keys fails at startup, the pod now exits instead of staying unhealthy. Connection errors to gitlab.com were resolved by a separate networking fix.

Response strategy: We first deployed a patch to remove in-memory caching of failed public key fetches, ensuring every request re-initialized the OIDC provider. After confirming improved reliability, we restored caching but changed the workflow so pods exit fatally if key fetch fails at startup. A separate networking issue causing connection failures was also fixed.

This ticket was created to track INC-8042, by incident.io 🔥