Log slow or rejected audit event deliveries and drop the per-event line

What does this MR do and why?

Logs a send to an external audit event streaming destination at info only when it is slow or rejected, and moves the per-event "Audit event streamed" line to debug.

BaseStreamer logs Audit event streamed at info level once per event per destination, which makes it by far the most frequent line the streaming code writes. Its only operational use was attributing a slow Sidekiq job to a namespace: join the slow jobs to those lines by correlation_id, read the destination off the line. That is how the slow tail was traced to S3 destinations on SDK default timeouts, and it is the only question the line answers. So instead of writing every send, this MR writes the ones worth looking at.

On both the Sidekiq path (BaseStreamer) and the NATS path (BatchedDispatcher), a send that takes 4 seconds or more or gets a non-2xx response is logged at info as Audit event delivery slow or rejected, with:

  • destination_id and destination_category;
  • gl_root_namespace_id (the standard Labkit field, so it lines up with the rest of the job's context);
  • duration_s (the standard Labkit::Fields::DURATION_S field, rounded to microseconds like the other durations we log);
  • status and error_category;
  • audit_event_id on the Sidekiq path and batch_size on the NATS path.

The threshold is SLOW_DELIVERY_SECONDS = 4, half the 8 second HTTP timeout, so a destination drifting towards the timeout is logged before it starts failing. The delivery result struct moves out of BaseStreamDestination into its own class, AuditEvents::Streaming::Outcome, built with keyword arguments, and gains status_code (from the response, or from a DestinationError), rejected?, slow_or_rejected? and logged_duration_s so both callers share the decision and log the same shape. The per-event line moves to debug and reuses the same destination fields.

Two notes on scope:

  • Gitlab::AppLogger takes its level from GITLAB_LOG_LEVEL and logs at debug when the variable is unset, so the volume reduction applies where the level is info or higher. Where it is not, the per-event line keeps being written and this MR adds the slow or rejected line on top.
  • !266404 (merged) (merged) makes a non-2xx response a failed delivery behind the audit_event_streaming_fail_on_non_2xx_response flag. With the flag on, a rejected send arrives as a DestinationError with no response: rejected? stays false, so the failure path logs it once and this line carries the slow tail only, with the status taken from the error. With the flag off, the rejected? branch here is the only record of a non-2xx.

References

Next in the stack: one delivery summary per job.

Screenshots or screen recordings

Not applicable, no UI changes.

How to set up and validate locally

  1. In a group with an Ultimate license, create an HTTP streaming destination (Secure > Audit events > Streams) pointing at a slow endpoint such as https://httpstat.us/200?sleep=5000.

  2. Trigger an audit event in the group through the UI (for example, change the group description) so the streaming worker runs in Sidekiq, then:

    grep 'Audit event delivery slow or rejected' log/application_json.log | tail -1 | jq '{destination_id, gl_root_namespace_id, duration_s, status, error_category}'

    Expected: duration_s around 5 and status 200.

  3. Change the destination URL to one that returns a non-2xx, for example https://httpstat.us/413, trigger another audit event and run the same grep. Expected: the same line with status 413 and a small duration_s.

  4. Point the destination at a fast endpoint that returns 200 (a https://webhook.site URL), trigger another audit event and confirm no new Audit event delivery slow or rejected line is written and that Audit event streamed no longer appears in log/application_json.log unless the log level is debug.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Vasyl Pedak

Merge request reports

Loading
Loading