Ack NATS audit batches when a destination fails permanently
What does this MR do and why?
On the NATS audit event streaming path, a group's messages are acked only when every destination succeeds. When one destination can never succeed (bad credentials, 401 or 403, 404, other 4xx, invalid configuration), JetStream redelivers the batch, and it is sent again to every destination, including the healthy ones. A group that leaves a broken destination in place alongside a working one gets the same events resent to the working one until max_deliver or the stream's 24h max_age. That wastes drainer time and HTTP calls, and duplicates events for the customer.
Behind the audit_event_streaming_nats_skip_permanent_failure_redelivery ops flag (default off), BatchedDispatcher now treats failures in invalid_config, auth, not_found and client_error as permanent:
- The failure is still recorded: it's logged and counted on the circuit breaker as before. The log now also has
redeliverandevent_count. - The dropped events are counted: a new counter,
gitlab_audit_event_streaming_nats_permanent_failure_events_total{destination_category, error_category}. - The batch is acked once every other destination succeeded or also failed permanently.
- Unchanged: temporary failures (
timeout,connection,server_error,rate_limited) and GitLab-side errors are redelivered as before.
A permanent category must not hide failures that a retry can fix, or events would be lost where they're retried today. The first commit fixes the mappings that did that:
| Failure | Before | After |
|---|---|---|
GCP token fetch that ran out of retries (Signet::AuthorizationError without a response) |
auth |
connection |
DNS lookup failed or timed out (BlockedUrlError caused by SocketError or Timeout::Error) |
invalid_config |
connection |
S3 RequestTimeout |
client_error |
timeout |
S3 OperationAborted, RequestTimeTooSkewed |
client_error, auth |
server_error |
| HTTP status 408 | client_error |
timeout |
| HTTP status 425 | client_error |
rate_limited |
A rejected token request (400, 401 or 403 from Google) still has a response and stays auth. A BlockedUrlError for any other reason stays invalid_config. On the Sidekiq path these mappings only change the error_category that is logged. Every destination-side category already counts towards the circuit breaker in the same way.
Production context (GitLab.com)
- Groups on NATS today (Wave 0, GCP only) have 2–10 group dispatches a day failing on a destination error, out of about 570k–720k. So the effect today is small.
- It grows with later waves: they bring HTTP groups with several destinations, including groups that have kept a broken destination alongside a working one.
- Once non-2xx responses count as failures (!266404 (merged)), about 4.0M HTTP responses a day become destination errors. Most of them are 401, 403, 415 and 404 from destinations that reject every send.
- Outside NATS: a destination with an open circuit breaker is already skipped without redelivery. This change reaches the same outcome straight away, without waiting for the breaker to open.
Verified end to end in GDK
The test group had a healthy HTTP destination plus one failing destination. In each case, one audit event was published through EnqueueService, partition 0 was drained, and it was drained again after the 30s ack deadline.
| Failing destination | Flag | Category | Result |
|---|---|---|---|
| GCP, malformed private key | off | invalid_config |
Redelivered; the healthy destination got a duplicate |
| GCP, malformed private key | on | invalid_config |
Acked; no duplicate; counter +1; logged redeliver: false |
| GCP, valid key for a service account that doesn't exist (real Google token endpoint) | on | auth |
Acked; no duplicate |
S3, invalid access key (real AWS InvalidAccessKeyId) |
on | auth |
Acked; no duplicate |
| HTTP, hostname that doesn't resolve | on | connection (temporary DNS) |
Redelivered, as before |
| HTTP, closed port | on | connection |
Redelivered, as before |
References
- NATS audit event streaming epic: gitlab-org#17582
- Limits redelivery to 10 deliveries: !267213 (merged)
- Counts non-2xx responses as failures: !266404 (merged)
Screenshots or screen recordings
Not applicable: no UI change.
How to set up and validate locally
-
In GDK, enable NATS and the
use_nats_for_audit_streamingapplication setting. Enableaudit_event_streaming_nats_consumerandaudit_event_streaming_via_nats(for your test group). -
On a top-level group, create two streaming destinations:
- an HTTP destination pointing at a local receiver that returns 200;
- a Google Cloud Logging destination with an invalid private key (any text that isn't a PEM key).
-
Trigger an audit event in the group, for example by changing its description, and let the drainer run (
AuditEvents::Streaming::NatsPartitionConsumerWorker). -
With the flag off, the HTTP receiver gets the event again about every 30 seconds, and
nats consumer info audit_events_streaming audit_streaming_consumer_0shows a message awaiting ack. -
Enable the flag:
Feature.enable(:audit_event_streaming_nats_skip_permanent_failure_redelivery) -
Trigger another event. The receiver gets it once, nothing stays awaiting ack, and
log/exceptions_json.logshowsOpenSSL::PKey::RSAErrorwith"extra.redeliver": false.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.