Provision NATS audit streaming consumers with bounded redelivery
What does this MR do and why?
Sets an explicit redelivery config on the NATS audit event streaming durable consumers, behind the audit_event_streaming_nats_consumer_redelivery ops flag (default off).
The durables are created implicitly by pull_subscribe, which never reapplies config to an existing consumer, so they run on the JetStream defaults: a 30s ack deadline and unlimited deliveries. A batch that takes longer than 30s is redelivered while a drainer is still sending it, and every group in that batch is sent twice. A batch that keeps failing is redelivered every 30s until the stream's 24h max_age drops it, with no signal.
NatsStreamProvisioner#ensure!now also ensures every partition's durable consumer, and corrects drift on its existing 5-minute, lease-guarded schedule.- With the flag on, consumers get:
ack_wait120s, above the drainer's 65s lease, so a batch still being sent is not redelivered to the next drainerbackoffof 2m, 5m, 15m, 30m and 1h; the last value repeatsmax_deliver: 10, about 6h of redelivery, well insidemax_age
- With the flag off, consumers get the JetStream defaults (30s, unlimited), so disabling the flag rolls back on the next provisioning run.
Gitlab::Nats::Clientgainsconsumer_info(returnsnilwhen the consumer is missing) andadd_consumer, which creates the consumer or updates it in place.NatsPartitionConsumerWorkerrecords each message whose last allowed delivery fails (num_deliveredat or abovemax_deliver, flag on), so we know exactly which message JetStream stopped redelivering:- It increments a new counter,
gitlab_audit_event_streaming_nats_max_deliveries_total, labelled byconsumer. - It logs one warning with
stream_sequence,audit_event_id,group_id,consumer,partition,num_deliveredand the outcome. It never logs the event body. - This is best-effort: an error here is tracked and never changes the dispatch outcome. The message stays unacked, as before.
- It increments a new counter,
Production state (GitLab.com)
| Value | |
|---|---|
| NATS server version | 2.15.0 on all 3 nodes (staging: 2.14.2) |
| Durable consumers | 257 (256 partitions plus the instance lane), replicated on 3 nodes |
| Consumer config | JetStream defaults: ack_wait 30s, max_deliver -1 |
| Max messages awaiting ack, last day | 77 |
| Redelivered messages, last day (consumer leaders) | 0 |
NatsConsumerWorker duration, last day |
1–5s for most runs |
- Deploy is a no-op with the flag off: the provisioner reads each consumer, sees the defaults, and changes nothing. The cost is 257 read-only consumer info requests every 5 minutes.
- After enabling: the next provisioning run updates all 257 durables in place. Nothing is recreated and delivery positions are kept.
Verified locally against NATS (GDK, 2.12.4)
- In place: existing durables created with the defaults update in place, and a second run changes nothing.
- Rollback: disabling the flag restores the defaults on the next run.
- At the limit: a message that reaches
max_deliverstops being redelivered and raises a$JS.EVENT.ADVISORY.CONSUMER.MAX_DELIVERIESadvisory. It can still be read by stream sequence untilmax_age. - Ack floor: it moves past that message once a later message is acked, so it does not hold up
num_pending.
Verified end to end in GDK: max deliveries
The test used a top-level group with an HTTP destination pointing at a closed local port, the flag on, and the partition consumer temporarily set to a 3s backoff with max_deliver: 10. One real audit event was published through EnqueueService and the drainer ran 14 times:
- Deliveries: exactly 10 failed sends, each
Errno::ECONNREFUSED, classified asconnection, so the outcome was:destination_error. - Counter:
0after runs 1–9,1from run 10 on. - Log: exactly one warning, on the 10th delivery:
{"message":"NATS audit event message failed on its last allowed delivery","stream_sequence":6,"audit_event_id":"<id>","group_id":<id>,"consumer":"audit_streaming_consumer_0","partition":"0","num_delivered":10,"outcome":"destination_error"}. - After the limit: no further deliveries in runs 11–14, nothing awaiting ack, and the message is still readable with
nats stream get audit_events_streaming 6.
Rollout and rollback
- Rollout: this is an
opsflag, so it has no rollout issue.- Enable it on staging, then check a consumer with
nats consumer info audit_events_streaming audit_streaming_consumer_0. - Then enable it on GitLab.com.
- Ship an alert on the max-deliveries advisory before or with enabling it, so messages that reach the limit are visible.
- Enable it on staging, then check a consumer with
- Rollback: disable the flag. The next provisioning run (within 5 minutes) restores the JetStream defaults.
- Known edge case: a message that has already been redelivered 10 or more times when the flag is enabled stops at its next redelivery.
- Messages that already hit the limit stay stopped after rollback. In GDK, after disabling the flag and restoring unlimited deliveries, a message that had reached
max_deliverwas not redelivered. It stays readable by stream sequence (from the warning) untilmax_age. - Alerting: the new counter's series for a consumer starts at 1, so the alert must also catch new series, not only
increase().
References
- NATS audit event streaming epic: gitlab-org#17582
- Related S3 timeout change: !267183 (merged)
Screenshots or screen recordings
Not applicable: no UI change.
How to set up and validate locally
-
Enable NATS in GDK (
gdk config set nats.enabled true && gdk reconfigure), enable theuse_nats_for_audit_streamingapplication setting, and letAuditEvents::Streaming::NatsConsumerWorkerrun once so the stream and durables exist. -
Check the current consumer config. It shows the defaults (
ack_wait30s,max_deliver-1, no backoff):nats consumer info audit_events_streaming audit_streaming_consumer_0 --json | jq '.config | {ack_wait, max_deliver, backoff}' -
In a Rails console, enable the flag and run the provisioner:
Feature.enable(:audit_event_streaming_nats_consumer_redelivery) AuditEvents::Streaming::NatsStreamProvisioner.new.ensure! -
Re-run the
nats consumer infocommand:ack_waitis 120000000000,max_deliveris 10, andbackofflists the five intervals in nanoseconds. -
Disable the flag, run
ensure!again, and confirm the defaults are back.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.