Enqueue to NATS in shadow mode and skip circuit breaker
What does this MR do and why?
Follow-up to !257611 (merged), which added a shadow mode to the NATS audit event streaming consumer.
In shadow mode we want the NATS consumer under real load without changing what the customer sees. This MR makes two changes, both gated behind the audit_event_streaming_nats_shadow_mode ops feature flag (default off, unreleased):
-
Enqueue to NATS in shadow mode (
EnqueueService): when shadow mode is on, publish the event to NATS and always deliver through Sidekiq. The customer stays on the live path and no event is lost, while the consumer gets real load. NATS publish is best-effort: failures are tracked and never block the Sidekiq delivery. Publish/fallback metrics are recorded so dashboards preview how NATS would perform as the real transport. -
Skip the circuit breaker in shadow dispatch (
BatchedDispatcher): shadow mode no longer callsCircuitBreaker.record_success. The breaker is keyed by destination and shared with the live Sidekiq path, so recording a success for a delivery that never happened would reset a real destination's failure state each batch. It now sleeps the simulated latency and returns:okwithout touching the breaker. Live delivery is unchanged.
References
- Follow-up to !257611 (merged).