Draft: Log one audit event delivery summary per job [skip ci]
What does this MR do and why?
Logs one summary of the audit event streaming deliveries per job, on both the Sidekiq path and the NATS path.
With the per-event "Audit event streamed" line gone (https://gitlab.com/gitlab-org/gitlab/-/merge_requests/https://gitlab.com/gitlab-org/gitlab/-/merge_requests/266796), a job's log says nothing about how many destinations it reached or how long the sends took unless one of them was slow or rejected. AuditEvents::AuditEventStreamingWorker and AuditEvents::Streaming::NatsPartitionConsumerWorker now log one audit_streaming_deliveries field through log_extra_metadata_on_done, next to the namespace_id the worker already logs:
"audit_streaming_deliveries": {
"http": { "sent": 3, "failed": 1, "max_duration_s": 0.42, "slowest_destination_id": 12 },
"aws": { "sent": 1, "failed": 0, "max_duration_s": 2.8, "slowest_destination_id": 7 }
}The category keys are the closed set http, aws, gcp, and the value types are fixed, so nothing is dynamic in the field names. sent and failed count deliver calls (one per event on the Sidekiq path, one per batch on the NATS path), max_duration_s is the slowest of them and slowest_destination_id the destination it went to. Both stay null until a duration has been measured; a delivery whose classification itself raised has none. The field is omitted when the job delivered nothing.
A small AuditEvents::Streaming::DeliverySummary accumulates it: each BaseStreamer and BatchedDispatcher records its own outcomes, ExternalDestinationStreamer merges the group and instance streamers, and the NATS consumer merges the dispatchers it ran in one drain. The summary adds roughly 150 bytes to each streaming job's done line, which is accepted against the per-event lines the previous MR removed. It complements the slow or rejected line, which still ties an individual slow send to its namespace, with an aggregate view of the job.
Stacked MR. This is based on https://gitlab.com/gitlab-org/gitlab/-/merge_requests/https://gitlab.com/gitlab-org/gitlab/-/merge_requests/266796 and targets its branch. It will be retargeted when the MR below it merges; until then the diff shows only this MR's commits.
References
- Implementation issue: https://gitlab.com/gitlab-org/gitlab/-/work_items/633532
- Base of the stack (error categories and the
record_deliveryhook): !262597 (merged) - Delivery outcome counter: !262609 (merged)
- Delivery duration histogram: !262610 (merged)
- Previous in the stack (slow or rejected delivery line): https://gitlab.com/gitlab-org/gitlab/-/merge_requests/https://gitlab.com/gitlab-org/gitlab/-/merge_requests/266796
Screenshots or screen recordings
Not applicable, no UI changes.
How to set up and validate locally
-
In a group with an Ultimate license, create an HTTP streaming destination (Secure > Audit events > Streams) pointing at a URL that returns 200, for example a https://webhook.site endpoint.
-
Trigger an audit event in the group through the UI (for example, change the group description) so the worker runs in Sidekiq, then read the job's done line:
grep audit_streaming_deliveries log/sidekiq.log | tail -1 | jq '{namespace_id, audit_event_type, audit_streaming_deliveries}'(
log/sidekiq_json.logon an Omnibus install.) Expected output:{ "namespace_id": 24, "audit_event_type": "group_description_updated", "audit_streaming_deliveries": { "http": { "sent": 1, "failed": 0, "max_duration_s": 0.213, "slowest_destination_id": 3 } } } -
Add a second destination pointing at
https://httpstat.us/500, trigger another audit event and run the same grep. Expected:"sent": 2, "failed": 1, withslowest_destination_idnaming whichever of the two took longer. -
Trigger an audit event in a group with no streaming destinations and confirm its done line has no
audit_streaming_deliveriesfield.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.