Add NATS audit streaming consumer workers

Huzaifa Iftikharrequested to merge
nats-08-consumer-workers into master

What does this MR do and why?

Add NATS audit streaming consumer workers

Two cron-driven workers that drain audit events from NATS and dispatch them to external destinations:

  • NatsConsumerWorker (1-minute cron scheduler): lease-guarded stream ensure, then fans out one drainer per partition. deduplicate :none so an arg-less scheduler is never suppressed by a constant idempotency key.
  • NatsPartitionConsumerWorker (drainer): owns one partition's durable pull consumer, drains in a loop within the minute, groups by top-level group, and dispatches via BatchedDispatcher. A per-partition ExclusiveLease gives mutual exclusion (not Sidekiq dedup). At-least-once ack after dispatch.

Closes - #604454 (closed) Part of epic &17582

References

Screenshots or screen recordings

Before After

How to set up and validate locally

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Merge request reports