Provision NATS audit streaming consumers with bounded redelivery

What does this MR do and why?

Sets an explicit redelivery config on the NATS audit event streaming durable consumers, behind the audit_event_streaming_nats_consumer_redelivery ops flag (default off).

The durables are created implicitly by pull_subscribe, which never reapplies config to an existing consumer, so they run on the JetStream defaults: a 30s ack deadline and unlimited deliveries. A batch that takes longer than 30s is redelivered while a drainer is still sending it, and every group in that batch is sent twice. A batch that keeps failing is redelivered every 30s until the stream's 24h max_age drops it, with no signal.

  • NatsStreamProvisioner#ensure! now also ensures every partition's durable consumer, and corrects drift on its existing 5-minute, lease-guarded schedule.
  • With the flag on, consumers get:
    • ack_wait 120s, above the drainer's 65s lease, so a batch still being sent is not redelivered to the next drainer
    • backoff of 2m, 5m, 15m, 30m and 1h; the last value repeats
    • max_deliver: 10, about 6h of redelivery, well inside max_age
  • With the flag off, consumers get the JetStream defaults (30s, unlimited), so disabling the flag rolls back on the next provisioning run.
  • Gitlab::Nats::Client gains consumer_info (returns nil when the consumer is missing) and add_consumer, which creates the consumer or updates it in place.
  • NatsPartitionConsumerWorker records each message whose last allowed delivery fails (num_delivered at or above max_deliver, flag on), so we know exactly which message JetStream stopped redelivering:
    • It increments a new counter, gitlab_audit_event_streaming_nats_max_deliveries_total, labelled by consumer.
    • It logs one warning with stream_sequence, audit_event_id, group_id, consumer, partition, num_delivered and the outcome. It never logs the event body.
    • This is best-effort: an error here is tracked and never changes the dispatch outcome. The message stays unacked, as before.
Production state (GitLab.com)
Value
NATS server version 2.15.0 on all 3 nodes (staging: 2.14.2)
Durable consumers 257 (256 partitions plus the instance lane), replicated on 3 nodes
Consumer config JetStream defaults: ack_wait 30s, max_deliver -1
Max messages awaiting ack, last day 77
Redelivered messages, last day (consumer leaders) 0
NatsConsumerWorker duration, last day 1–5s for most runs
  • Deploy is a no-op with the flag off: the provisioner reads each consumer, sees the defaults, and changes nothing. The cost is 257 read-only consumer info requests every 5 minutes.
  • After enabling: the next provisioning run updates all 257 durables in place. Nothing is recreated and delivery positions are kept.
Verified locally against NATS (GDK, 2.12.4)
  • In place: existing durables created with the defaults update in place, and a second run changes nothing.
  • Rollback: disabling the flag restores the defaults on the next run.
  • At the limit: a message that reaches max_deliver stops being redelivered and raises a $JS.EVENT.ADVISORY.CONSUMER.MAX_DELIVERIES advisory. It can still be read by stream sequence until max_age.
  • Ack floor: it moves past that message once a later message is acked, so it does not hold up num_pending.
Verified end to end in GDK: max deliveries

The test used a top-level group with an HTTP destination pointing at a closed local port, the flag on, and the partition consumer temporarily set to a 3s backoff with max_deliver: 10. One real audit event was published through EnqueueService and the drainer ran 14 times:

  • Deliveries: exactly 10 failed sends, each Errno::ECONNREFUSED, classified as connection, so the outcome was :destination_error.
  • Counter: 0 after runs 1–9, 1 from run 10 on.
  • Log: exactly one warning, on the 10th delivery: {"message":"NATS audit event message failed on its last allowed delivery","stream_sequence":6,"audit_event_id":"<id>","group_id":<id>,"consumer":"audit_streaming_consumer_0","partition":"0","num_delivered":10,"outcome":"destination_error"}.
  • After the limit: no further deliveries in runs 11–14, nothing awaiting ack, and the message is still readable with nats stream get audit_events_streaming 6.
Rollout and rollback
  • Rollout: this is an ops flag, so it has no rollout issue.
    1. Enable it on staging, then check a consumer with nats consumer info audit_events_streaming audit_streaming_consumer_0.
    2. Then enable it on GitLab.com.
    3. Ship an alert on the max-deliveries advisory before or with enabling it, so messages that reach the limit are visible.
  • Rollback: disable the flag. The next provisioning run (within 5 minutes) restores the JetStream defaults.
  • Known edge case: a message that has already been redelivered 10 or more times when the flag is enabled stops at its next redelivery.
  • Messages that already hit the limit stay stopped after rollback. In GDK, after disabling the flag and restoring unlimited deliveries, a message that had reached max_deliver was not redelivered. It stays readable by stream sequence (from the warning) until max_age.
  • Alerting: the new counter's series for a consumer starts at 1, so the alert must also catch new series, not only increase().

References

Screenshots or screen recordings

Not applicable: no UI change.

How to set up and validate locally

  1. Enable NATS in GDK (gdk config set nats.enabled true && gdk reconfigure), enable the use_nats_for_audit_streaming application setting, and let AuditEvents::Streaming::NatsConsumerWorker run once so the stream and durables exist.

  2. Check the current consumer config. It shows the defaults (ack_wait 30s, max_deliver -1, no backoff):

    nats consumer info audit_events_streaming audit_streaming_consumer_0 --json | jq '.config | {ack_wait, max_deliver, backoff}'
  3. In a Rails console, enable the flag and run the provisioner:

    Feature.enable(:audit_event_streaming_nats_consumer_redelivery)
    AuditEvents::Streaming::NatsStreamProvisioner.new.ensure!
  4. Re-run the nats consumer info command: ack_wait is 120000000000, max_deliver is 10, and backoff lists the five intervals in nanoseconds.

  5. Disable the flag, run ensure! again, and confirm the defaults are back.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Huzaifa Iftikhar

Merge request reports

Loading
Loading