Decouple NATS shadow publish from shadow drain

Huzaifa Iftikharrequested to merge
nats-shadow-mode-decouple-publish into master

What does this MR do and why?

Follow-up to !257803 (merged). Adds a new ops feature flag audit_event_streaming_nats_shadow_publish (default off, unreleased) that gates only the publish side of NATS shadow mode.

Today audit_event_streaming_nats_shadow_mode gates two things at once: whether EnqueueService publishes to NATS, and whether BatchedDispatcher discards instead of delivering. Because they share one flag, there is no clean way to stop feeding the stream while still letting the consumer drain the existing backlog. That forces a cutover that relies on flag ordering or the 24h max_age expiry.

This MR splits the publish decision out. EnqueueService#shadow_mode? now also requires audit_event_streaming_nats_shadow_publish; the consumer's discard path (BatchedDispatcher) is unchanged and still keyed to audit_event_streaming_nats_shadow_mode.

Clean cutover this enables

  1. Disable audit_event_streaming_nats_shadow_publish — publishing to NATS stops immediately; customers stay fully served by Sidekiq. No new messages enter the stream.
  2. Leave audit_event_streaming_nats_shadow_mode and the consumer on — drainers keep pulling the existing backlog, sleep + discard + ack. The stream drains to empty with no delivery to customer endpoints; record_stream_depth reaching 0 signals completion.
  3. Disable audit_event_streaming_nats_shadow_mode (and the consumer) once the stream is empty.

No reliance on max_age expiry and no window where a pending message could be delivered live.

Changes

  • New ops flag audit_event_streaming_nats_shadow_publish.
  • One added condition in EnqueueService#shadow_mode?.
  • Spec: existing shadow-mode specs default the new flag on (behaviour unchanged); new context covers publish-off still delivering via Sidekiq and never publishing.

References

Merge request reports