Count audit event deliveries skipped by an open circuit breaker
What does this MR do and why?
Adds gitlab_audit_event_streaming_circuit_breaker_deliveries_skipped_total, a counter of audit events not sent because the destination's circuit breaker was open, labelled with feature_category, scope (group, instance), destination_category (http, aws, gcp) and delivery_mode (per_event, batched). It also documents the existing gitlab_audit_event_streaming_circuit_breaker_total (destinations tripped open), which has been emitted since 19.0 but had no row in the metrics table.
The delivery counter from !262609 (merged) only sees deliveries that were attempted. While a destination's circuit is open, AuditEvents::Streaming::CircuitBreaker.reject_open drops it before any delivery happens, so a customer whose destination is skipped for five minutes at a time is invisible in both the outcome counter and the duration histogram. This counter closes that gap:
reject_openincrements it for every destination it filters out, by the number of audit events the caller was about to send: one on the Sidekiq per-event path, the batch size on the NATS path. Counting events rather than calls keeps the unit the same asgitlab_audit_event_streaming_deliveries_total, so skipped divided by delivered plus skipped is a meaningful ratio on both paths.delivery_modetells the two paths apart.- The breaker runs before event and namespace filters, so the counter can exceed the deliveries the filters would have let through. While the NATS shadow mode flag is on both paths run and skips count twice.
- The increment is guarded with
rescue StandardErrorand reported throughGitlab::ErrorTracking.track_exception, matching therecord_deliveryguards inBaseStreamDestination, so a metrics failure cannot change which destinations are returned. - Every scope, category and delivery mode series is pre-created at zero in
initialize_slis!on Sidekiq (12 series). - The docs row of the existing
gitlab_audit_event_streaming_circuit_breaker_totalnow explains that it counts the breaker opening, not the deliveries skipped while it is open, and points at the new counter.
This MR was stacked on !266796 (merged), which has merged; it now targets master directly.
References
- Implementation issue: https://gitlab.com/gitlab-org/gitlab/-/work_items/633532
- Base of the stack (error categories and the
record_deliveryhook): !262597 (merged) - Slow or rejected delivery log line (merged): !266796 (merged)
- Delivery duration histogram: !262610 (merged)
- Delivery outcome counter: !262609 (merged)
- First attempt this replaces: !261765 (closed)
Screenshots or screen recordings
Not applicable, no UI changes.
How to set up and validate locally
Metrics are per process, so the quickest check trips the breaker and runs the streaming worker inline in a Rails console, then reads the counter from the same process.
-
In a group with an Ultimate license, create an HTTP streaming destination (Secure > Audit events > Streams) pointing at any URL.
-
In
bin/rails console:destination = AuditEvents::Group::ExternalStreamingDestination.where(group_id: <group_id>).order(:id).last AuditEvents::Streaming::CircuitBreaker::FAILURE_THRESHOLD.times do AuditEvents::Streaming::CircuitBreaker.record_failure(destination) end AuditEvents::Streaming::CircuitBreaker.open?(destination) # => true event = AuditEvent.where(entity_id: <group_id>).order(:id).last AuditEvents::AuditEventStreamingWorker.new.perform('audit_operation', event.id) counter = Gitlab::Metrics::AuditEventStreamingSlis.deliveries_skipped_counter counter.get(feature_category: :audit_events, scope: 'group', destination_category: 'http', delivery_mode: 'per_event') # => 1.0The delivery counter from the previous MR does not move, since nothing was attempted. The open key expires after five minutes; delete it early with
Gitlab::Redis::SharedState.with { |r| r.del("audit_events:streaming:circuit:{group}:open:#{destination.id}") }. -
To see the series scraped by Prometheus instead, enable the GDK Prometheus stack:
gdk config set prometheus.enabled true gdk reconfigure gdk restartTrip the breaker as above, trigger an audit event in the group through the UI, open
http://gdk.test:9090(orcurl http://gdk.test:3807/metricsfor the Sidekiq exporter directly) and querygitlab_audit_event_streaming_circuit_breaker_deliveries_skipped_total.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.