Count audit event streaming deliveries by outcome and error category

What does this MR do and why?

Adds gitlab_audit_event_streaming_deliveries_total, a counter of deliveries to external streaming destinations by outcome (one per audit event per destination per attempt, so a NATS redelivery counts again and the total is not unique audit events), labelled with destination_category (http, aws, gcp), outcome (success, internal_error for a GitLab-side failure, destination_error for a destination-side one) and error_category (none on success, otherwise one of the categories BaseStreamDestination::ERROR_CATEGORIES defines).

Today the only delivery signal is gitlab_audit_event_streaming_nats_dispatch_destination_error_total. It covers the NATS path only, has no breakdown by destination type, and cannot tell a timeout from a rejected credential. Since !262597 (merged) both the Sidekiq path (BaseStreamer) and the NATS path (BatchedDispatcher) deliver through BaseStreamDestination#deliver / #deliver_batch, which call a private record_delivery(outcome, count) hook after the outcome is settled. This MR fills in that hook:

  • BaseStreamDestination#record_delivery increments the counter by the event count under the destination's category, the outcome and the error category. The outcome is success when the event was delivered, internal_error when the outcome's category is internal_error, and destination_error otherwise. The base's with_outcome already rescues and reports a failure in the hook, so a metrics failure can never change a delivery outcome.
  • Gitlab::Metrics::AuditEventStreamingSlis defines the counter next to the existing NATS metrics and pre-creates every series at zero in initialize_slis! (each category in Destinations::BY_CATEGORY times success/none, internal_error/internal_error, and destination_error times every destination-side error category), so rate() panels read zero on a healthy fleet instead of "no data".
  • The NATS destination error counter is kept and documented as deprecated in favour of this one.

This is a plain counter, not a gitlab_sli_* series: a failing customer destination is degradation, not a GitLab error budget signal.

This replaces the first attempt in !261765 (closed), which recorded at each call site. Response status inspection (counting non-2xx responses as failures) is out of scope for this stack.

Stacked MR. This is based on !262597 (merged) and targets its branch. It will be retargeted to master once that MR merges; until then the diff shows only this MR's commit.

References

Next in the stack: a delivery duration histogram, then a circuit breaker skipped counter.

Screenshots or screen recordings

Not applicable, no UI changes.

How to set up and validate locally

Metrics are per process, so the quickest check runs the streaming worker inline in a Rails console and reads the counter from the same process.

  1. In a group with an Ultimate license, create an HTTP streaming destination (Secure > Audit events > Streams). Point it at a URL that returns 200 first, for example a https://webhook.site endpoint.

  2. In bin/rails console, with an audit event the group already has (any change in the group's settings creates one):

    Gitlab::Metrics::AuditEventStreamingSlis.initialize_slis!
    event = AuditEvents::GroupAuditEvent.by_group(<group_id>).order(:id).last
    AuditEvents::AuditEventStreamingWorker.new.perform(event.event_name, event.id, nil, 'AuditEvents::GroupAuditEvent')
    
    counter = Gitlab::Metrics::AuditEventStreamingSlis.delivery_counter
    counter.get(feature_category: :audit_events, category: 'http', outcome: :success, error_category: :none)
    # => 1.0
  3. Change the destination URL to an unreachable host such as https://localhost:9, run the perform line again, then:

    counter.get(feature_category: :audit_events, category: 'http', outcome: :destination_error, error_category: :connection)
    # => 1.0
  4. To see the series scraped by Prometheus instead, enable the GDK Prometheus stack:

    gdk config set prometheus.enabled true
    gdk reconfigure
    gdk restart

    Trigger an audit event in the group through the UI and open http://gdk.test:9090 (or curl http://gdk.test:3807/metrics for the Sidekiq exporter directly) and query gitlab_audit_event_streaming_deliveries_total. Every category, outcome and error category series is present at 0 before any delivery happens.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Vasyl Pedak

Merge request reports

Loading
Loading