Ack NATS audit batches when a destination fails permanently

What does this MR do and why?

On the NATS audit event streaming path, a group's messages are acked only when every destination succeeds. When one destination can never succeed (bad credentials, 401 or 403, 404, other 4xx, invalid configuration), JetStream redelivers the batch, and it is sent again to every destination, including the healthy ones. A group that leaves a broken destination in place alongside a working one gets the same events resent to the working one until max_deliver or the stream's 24h max_age. That wastes drainer time and HTTP calls, and duplicates events for the customer.

Behind the audit_event_streaming_nats_skip_permanent_failure_redelivery ops flag (default off), BatchedDispatcher now treats failures in invalid_config, auth, not_found and client_error as permanent:

  • The failure is still recorded: it's logged and counted on the circuit breaker as before. The log now also has redeliver and event_count.
  • The dropped events are counted: a new counter, gitlab_audit_event_streaming_nats_permanent_failure_events_total{destination_category, error_category}.
  • The batch is acked once every other destination succeeded or also failed permanently.
  • Unchanged: temporary failures (timeout, connection, server_error, rate_limited) and GitLab-side errors are redelivered as before.

A permanent category must not hide failures that a retry can fix, or events would be lost where they're retried today. The first commit fixes the mappings that did that:

Failure Before After
GCP token fetch that ran out of retries (Signet::AuthorizationError without a response) auth connection
DNS lookup failed or timed out (BlockedUrlError caused by SocketError or Timeout::Error) invalid_config connection
S3 RequestTimeout client_error timeout
S3 OperationAborted, RequestTimeTooSkewed client_error, auth server_error
HTTP status 408 client_error timeout
HTTP status 425 client_error rate_limited

A rejected token request (400, 401 or 403 from Google) still has a response and stays auth. A BlockedUrlError for any other reason stays invalid_config. On the Sidekiq path these mappings only change the error_category that is logged. Every destination-side category already counts towards the circuit breaker in the same way.

Production context (GitLab.com)
  • Groups on NATS today (Wave 0, GCP only) have 2–10 group dispatches a day failing on a destination error, out of about 570k–720k. So the effect today is small.
  • It grows with later waves: they bring HTTP groups with several destinations, including groups that have kept a broken destination alongside a working one.
  • Once non-2xx responses count as failures (!266404 (merged)), about 4.0M HTTP responses a day become destination errors. Most of them are 401, 403, 415 and 404 from destinations that reject every send.
  • Outside NATS: a destination with an open circuit breaker is already skipped without redelivery. This change reaches the same outcome straight away, without waiting for the breaker to open.
Verified end to end in GDK

The test group had a healthy HTTP destination plus one failing destination. In each case, one audit event was published through EnqueueService, partition 0 was drained, and it was drained again after the 30s ack deadline.

Failing destination Flag Category Result
GCP, malformed private key off invalid_config Redelivered; the healthy destination got a duplicate
GCP, malformed private key on invalid_config Acked; no duplicate; counter +1; logged redeliver: false
GCP, valid key for a service account that doesn't exist (real Google token endpoint) on auth Acked; no duplicate
S3, invalid access key (real AWS InvalidAccessKeyId) on auth Acked; no duplicate
HTTP, hostname that doesn't resolve on connection (temporary DNS) Redelivered, as before
HTTP, closed port on connection Redelivered, as before

References

Screenshots or screen recordings

Not applicable: no UI change.

How to set up and validate locally

  1. In GDK, enable NATS and the use_nats_for_audit_streaming application setting. Enable audit_event_streaming_nats_consumer and audit_event_streaming_via_nats (for your test group).

  2. On a top-level group, create two streaming destinations:

    • an HTTP destination pointing at a local receiver that returns 200;
    • a Google Cloud Logging destination with an invalid private key (any text that isn't a PEM key).
  3. Trigger an audit event in the group, for example by changing its description, and let the drainer run (AuditEvents::Streaming::NatsPartitionConsumerWorker).

  4. With the flag off, the HTTP receiver gets the event again about every 30 seconds, and nats consumer info audit_events_streaming audit_streaming_consumer_0 shows a message awaiting ack.

  5. Enable the flag:

    Feature.enable(:audit_event_streaming_nats_skip_permanent_failure_redelivery)
  6. Trigger another event. The receiver gets it once, nothing stays awaiting ack, and log/exceptions_json.log shows OpenSSL::PKey::RSAError with "extra.redeliver": false.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Merge request reports

Loading
Loading