Record audit event streaming delivery duration per destination type
What does this MR do and why?
Adds gitlab_audit_event_streaming_delivery_duration_seconds, a histogram of the wall time of each delivery call to an external streaming destination (the whole stream or stream_batch call, so payload serialization is included), labelled with category (http, aws, gcp).
The delivery outcome counter from !262609 (merged) tells us how many deliveries fail and why (outcome is success, destination_error or internal_error, error_category names the failure), but not how slow the ones that succeed are, and a destination that is about to start timing out shows up first as a drift towards the top buckets. BaseStreamDestination#with_outcome already wraps every stream / stream_batch call on both the Sidekiq and NATS paths, so this MR:
- measures the call with
Gitlab::Metrics::System.monotonic_timeinsidewith_outcomeand stores the elapsed seconds on theOutcomeasduration_s, so therecord_deliveryhook and later callers can both read it; - observes it on the histogram next to the counter increment. The hook runs inside the guard
with_outcomealready has, so a metrics failure still cannot change the delivery outcome. When classification itself raised there is no duration, and the observation is skipped instead of raising on nil; - uses buckets
0.05, 0.1, 0.25, 0.5, 1, 2, 4, 8, 16, 30, 60, 120, 300seconds. The HTTP destination times out at 8 seconds per open/read/write phase, so the buckets are dense up to 16 seconds and a slow webhook drifts into them before it fails. The buckets from 30 to 300 seconds are there for S3 and GCP, which still run on their client defaults: today's tail sits between 30 seconds and about five minutes, and without them it would all land in+Inf, so we could not get a baseline before cutting those timeouts to 8 seconds in https://gitlab.com/gitlab-org/gitlab/-/work_items/636583.
The histogram is a plain metric, not a gitlab_sli_* apdex: destination latency is the customer's infrastructure, not a GitLab error budget signal.
Stacked MR. This is based on !262609 (merged) and targets its branch. It will be retargeted when the MR below it merges; until then the diff shows only this MR's commits.
References
- Implementation issue: https://gitlab.com/gitlab-org/gitlab/-/work_items/633532
- Base of the stack (error categories and the
record_deliveryhook): !262597 (merged) - Previous in the stack (delivery outcome counter): !262609 (merged)
- S3 and GCP timeouts to be cut to 8 seconds: https://gitlab.com/gitlab-org/gitlab/-/work_items/636583
- First attempt this replaces: !261765 (closed)
Next in the stack: the slow or rejected delivery log line, then the per-job delivery summary.
Screenshots or screen recordings
Not applicable, no UI changes.
How to set up and validate locally
Metrics are per process, so the quickest check runs the streaming worker inline in a Rails console and reads the histogram from the same process.
-
In a group with an Ultimate license, create an HTTP streaming destination (Secure > Audit events > Streams) pointing at a URL that returns 200, for example a https://webhook.site endpoint.
-
In
bin/rails console:Gitlab::Metrics::AuditEventStreamingSlis.initialize_slis! event = AuditEvent.where(entity_id: <group_id>).order(:id).last AuditEvents::AuditEventStreamingWorker.new.perform('audit_operation', event.id) histogram = Gitlab::Metrics::AuditEventStreamingSlis.delivery_duration_histogram histogram.get(feature_category: :audit_events, category: 'http') # => {"0.05"=>0.0, "0.1"=>0.0, "0.25"=>1.0, "0.5"=>1.0, ..., "300"=>1.0, "+Inf"=>1.0, "sum"=>0.21...}The bucket at which the cumulative count becomes 1 is the request's duration.
-
To see the series scraped by Prometheus instead, enable the GDK Prometheus stack:
gdk config set prometheus.enabled true gdk reconfigure gdk restartTrigger an audit event in the group through the UI (for example, change the group description) so the worker runs in Sidekiq, then open
http://gdk.test:9090(orcurl http://gdk.test:3807/metrics | grep gitlab_audit_event_streaming_delivery_duration_secondsfor the Sidekiq exporter directly) and queryhistogram_quantile(0.95, rate(gitlab_audit_event_streaming_delivery_duration_seconds_bucket[5m])).
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.