Count audit event deliveries skipped by an open circuit breaker

What does this MR do and why?

Adds gitlab_audit_event_streaming_circuit_breaker_deliveries_skipped_total, a counter of audit events not sent because the destination's circuit breaker was open, labelled with feature_category, scope (group, instance), destination_category (http, aws, gcp) and delivery_mode (per_event, batched). It also documents the existing gitlab_audit_event_streaming_circuit_breaker_total (destinations tripped open), which has been emitted since 19.0 but had no row in the metrics table.

The delivery counter from !262609 (merged) only sees deliveries that were attempted. While a destination's circuit is open, AuditEvents::Streaming::CircuitBreaker.reject_open drops it before any delivery happens, so a customer whose destination is skipped for five minutes at a time is invisible in both the outcome counter and the duration histogram. This counter closes that gap:

  • reject_open increments it for every destination it filters out, by the number of audit events the caller was about to send: one on the Sidekiq per-event path, the batch size on the NATS path. Counting events rather than calls keeps the unit the same as gitlab_audit_event_streaming_deliveries_total, so skipped divided by delivered plus skipped is a meaningful ratio on both paths. delivery_mode tells the two paths apart.
  • The breaker runs before event and namespace filters, so the counter can exceed the deliveries the filters would have let through. While the NATS shadow mode flag is on both paths run and skips count twice.
  • The increment is guarded with rescue StandardError and reported through Gitlab::ErrorTracking.track_exception, matching the record_delivery guards in BaseStreamDestination, so a metrics failure cannot change which destinations are returned.
  • Every scope, category and delivery mode series is pre-created at zero in initialize_slis! on Sidekiq (12 series).
  • The docs row of the existing gitlab_audit_event_streaming_circuit_breaker_total now explains that it counts the breaker opening, not the deliveries skipped while it is open, and points at the new counter.

This MR was stacked on !266796 (merged), which has merged; it now targets master directly.

References

Screenshots or screen recordings

Not applicable, no UI changes.

How to set up and validate locally

Metrics are per process, so the quickest check trips the breaker and runs the streaming worker inline in a Rails console, then reads the counter from the same process.

  1. In a group with an Ultimate license, create an HTTP streaming destination (Secure > Audit events > Streams) pointing at any URL.

  2. In bin/rails console:

    destination = AuditEvents::Group::ExternalStreamingDestination.where(group_id: <group_id>).order(:id).last
    AuditEvents::Streaming::CircuitBreaker::FAILURE_THRESHOLD.times do
      AuditEvents::Streaming::CircuitBreaker.record_failure(destination)
    end
    AuditEvents::Streaming::CircuitBreaker.open?(destination)
    # => true
    
    event = AuditEvent.where(entity_id: <group_id>).order(:id).last
    AuditEvents::AuditEventStreamingWorker.new.perform('audit_operation', event.id)
    
    counter = Gitlab::Metrics::AuditEventStreamingSlis.deliveries_skipped_counter
    counter.get(feature_category: :audit_events, scope: 'group', destination_category: 'http', delivery_mode: 'per_event')
    # => 1.0

    The delivery counter from the previous MR does not move, since nothing was attempted. The open key expires after five minutes; delete it early with Gitlab::Redis::SharedState.with { |r| r.del("audit_events:streaming:circuit:{group}:open:#{destination.id}") }.

  3. To see the series scraped by Prometheus instead, enable the GDK Prometheus stack:

    gdk config set prometheus.enabled true
    gdk reconfigure
    gdk restart

    Trip the breaker as above, trigger an audit event in the group through the UI, open http://gdk.test:9090 (or curl http://gdk.test:3807/metrics for the Sidekiq exporter directly) and query gitlab_audit_event_streaming_circuit_breaker_deliveries_skipped_total.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Vasyl Pedak

Merge request reports

Loading
Loading