Count audit event streaming deliveries by outcome and error category
What does this MR do and why?
Adds gitlab_audit_event_streaming_deliveries_total, a counter of deliveries to external streaming destinations by outcome (one per audit event per destination per attempt, so a NATS redelivery counts again and the total is not unique audit events), labelled with destination_category (http, aws, gcp), outcome (success, internal_error for a GitLab-side failure, destination_error for a destination-side one) and error_category (none on success, otherwise one of the categories BaseStreamDestination::ERROR_CATEGORIES defines).
Today the only delivery signal is gitlab_audit_event_streaming_nats_dispatch_destination_error_total. It covers the NATS path only, has no breakdown by destination type, and cannot tell a timeout from a rejected credential. Since !262597 (merged) both the Sidekiq path (BaseStreamer) and the NATS path (BatchedDispatcher) deliver through BaseStreamDestination#deliver / #deliver_batch, which call a private record_delivery(outcome, count) hook after the outcome is settled. This MR fills in that hook:
BaseStreamDestination#record_deliveryincrements the counter by the event count under the destination's category, the outcome and the error category. The outcome issuccesswhen the event was delivered,internal_errorwhen the outcome's category isinternal_error, anddestination_errorotherwise. The base'swith_outcomealready rescues and reports a failure in the hook, so a metrics failure can never change a delivery outcome.Gitlab::Metrics::AuditEventStreamingSlisdefines the counter next to the existing NATS metrics and pre-creates every series at zero ininitialize_slis!(each category inDestinations::BY_CATEGORYtimessuccess/none,internal_error/internal_error, anddestination_errortimes every destination-side error category), sorate()panels read zero on a healthy fleet instead of "no data".- The NATS destination error counter is kept and documented as deprecated in favour of this one.
This is a plain counter, not a gitlab_sli_* series: a failing customer destination is degradation, not a GitLab error budget signal.
This replaces the first attempt in !261765 (closed), which recorded at each call site. Response status inspection (counting non-2xx responses as failures) is out of scope for this stack.
Stacked MR. This is based on !262597 (merged) and targets its branch. It will be retargeted to master once that MR merges; until then the diff shows only this MR's commit.
References
- Implementation issue: https://gitlab.com/gitlab-org/gitlab/-/work_items/633532
- Base of the stack (error categories and the
record_deliveryhook): !262597 (merged) - First attempt this replaces: !261765 (closed)
Next in the stack: a delivery duration histogram, then a circuit breaker skipped counter.
Screenshots or screen recordings
Not applicable, no UI changes.
How to set up and validate locally
Metrics are per process, so the quickest check runs the streaming worker inline in a Rails console and reads the counter from the same process.
-
In a group with an Ultimate license, create an HTTP streaming destination (Secure > Audit events > Streams). Point it at a URL that returns 200 first, for example a https://webhook.site endpoint.
-
In
bin/rails console, with an audit event the group already has (any change in the group's settings creates one):Gitlab::Metrics::AuditEventStreamingSlis.initialize_slis! event = AuditEvents::GroupAuditEvent.by_group(<group_id>).order(:id).last AuditEvents::AuditEventStreamingWorker.new.perform(event.event_name, event.id, nil, 'AuditEvents::GroupAuditEvent') counter = Gitlab::Metrics::AuditEventStreamingSlis.delivery_counter counter.get(feature_category: :audit_events, category: 'http', outcome: :success, error_category: :none) # => 1.0 -
Change the destination URL to an unreachable host such as
https://localhost:9, run theperformline again, then:counter.get(feature_category: :audit_events, category: 'http', outcome: :destination_error, error_category: :connection) # => 1.0 -
To see the series scraped by Prometheus instead, enable the GDK Prometheus stack:
gdk config set prometheus.enabled true gdk reconfigure gdk restartTrigger an audit event in the group through the UI and open
http://gdk.test:9090(orcurl http://gdk.test:3807/metricsfor the Sidekiq exporter directly) and querygitlab_audit_event_streaming_deliveries_total. Every category, outcome and error category series is present at 0 before any delivery happens.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.