Let each streaming destination classify its own delivery errors

What does this MR do and why?

Changes the structure of audit event streaming delivery so that delivery outcome tracking can be added on top of it. No metrics are added here; the follow-up MRs are listed under References.

The main decision is to encapsulate error classification within each stream destination. Until now a single ErrorClassifier module held every destination's SDK exceptions in one list (AWS service and networking errors, GCP auth and key errors, the HTTP client's transport errors) and answered one boolean, log or Sentry. Adding a destination meant editing that shared list, and both delivery paths re-checked the exception to make the same decision. Azure Blob, Splunk, Datadog and Snowflake destinations are all in the backlog, so this would keep growing.

After this MR:

  • BaseStreamDestination owns the fixed list of error categories (timeout, connection, auth, not_found, rate_limited, client_error, server_error, invalid_config, internal_error) and the transport mapping that every destination shares, since HTTP and GCP both use Gitlab::HTTP and S3 errors unwrap to the same socket exceptions.
  • AmazonS3StreamDestination#classify_error maps the AWS SDK: service errors by HTTP status, SlowDown (the only throttling code S3 returns) to rate_limited, PermanentRedirect (bucket in another region) to invalid_config, networking errors by their wrapped cause, missing credentials, invalid region. Then falls back to the shared mapping.
  • GoogleCloudLoggingStreamDestination#classify_error maps Signet authorisation and RSA key errors and falls back to the shared mapping. These two cover everything the GCP client stack raises for a bad configuration: googleauth retries every token fetch failure and wraps it in Signet::AuthorizationError, and a malformed PEM raises OpenSSL::PKey::RSAError. Anything else from the GCP adapter is a GitLab bug and now reaches Sentry, where master's catch-all logged it as customer configuration.
  • HttpStreamDestination uses the shared mapping unchanged.
  • BaseStreamDestination#deliver and #deliver_batch wrap the subclass's stream calls and never raise. They return an Outcome with the error and its category, and call a private no-op record_delivery(outcome, count) hook where the metrics will plug in. The hook call is wrapped in a rescue, so a bug in it is reported to Sentry without changing the outcome. A category outside ERROR_CATEGORIES is treated as internal_error, so a subclass bug cannot pass a GitLab error off as a customer one.
  • BaseStreamer (Sidekiq path) and BatchedDispatcher (NATS path) branch on the outcome (outcome.internal_error?) instead of rescuing and classifying themselves. Classification happens once, in the destination. Their remaining rescue only covers errors raised outside the delivery call, such as internal event tracking, an unknown category, or the success and failure handlers themselves, which are GitLab-side by definition.
  • ErrorClassifier and the log_only_errors hook are removed. The category-to-adapter registry becomes Destinations.for_category, next to the adapters, and both callers use it instead of reaching into a sibling's constant.
  • Local variables and spec doubles that hold adapter instances are named stream_destination rather than streamer, so they are not confused with the real BaseStreamer.

Logging and error tracking cleanup that rides along:

  • Failure log context now carries error_category, so Kibana can break failures down by cause without confusing it with exception.class.

Routing changes worth a reviewer's eye, all a consequence of classifying per destination:

  • Seahorse::Client::NetworkingError (S3 unreachable) and Aws::Errors::MissingCredentialsError previously fell through to Sentry with no circuit breaker count. They are now destination-side (connection and auth), so they are logged and count toward the breaker.
  • Non-S3 Aws::Errors::ServiceError subclasses are now destination-side too. Master only matched the S3 subclass, which is the only one the code can raise today.
  • For GCP destinations, master's log_only_errors = [StandardError] made every error log-only, including GitLab bugs inside the adapter (a NoMethodError in payload building, an oversized event hitting the JSON size limit) and errors raised around the delivery call (internal event tracking, an unknown category, building the batch). All of those are now internal_error: Sentry, and failed on the NATS path so the batch is retried. Only Signet and RSA key errors stay destination-side for GCP.
  • OpenSSL::PKey::RSAError and Signet::AuthorizationError are only classified by the GCP destination. An HTTP or S3 destination raising them, which cannot happen with the current clients, would now be internal_error.

References

Stacked follow-ups

These Draft MRs target this branch and retarget to master when it merges. The metrics MRs plug into the record_delivery hook and are stacked on each other in this order:

  1. !262609 (merged) adds gitlab_audit_event_streaming_delivery_events_total with category, outcome (success, internal_error, destination_error) and error_category labels. It replaces the first attempt in !261765 (closed).
  2. !262610 (merged) adds gitlab_audit_event_streaming_delivery_duration_seconds per category, with buckets up to 300s so the current S3 and GCP tail can be baselined before their timeouts are cut to 8s (https://gitlab.com/gitlab-org/gitlab/-/work_items/636583).
  3. !262612 (merged) adds gitlab_audit_event_streaming_circuit_breaker_skipped_total for attempts skipped while a destination's circuit is open.
  4. !266796 (merged) logs an info line for sends that take 4s or more or get a non-2xx response, with destination, root namespace, duration and status, and moves the per-event "Audit event streamed" line to debug.
  5. !266797 (closed) logs one audit_streaming_deliveries summary field per job on the streaming worker and the NATS consumer.

Beside the metrics stack:

  • !266404 (merged) makes a non-2xx response from an HTTP or GCP destination a classified failure. Today both adapters return the response without checking the status, so a rejected write counts as delivered; this MR does not change that, the follow-up does.

Screenshots or screen recordings

Not applicable, no UI changes.

How to set up and validate locally

  1. In a group with an Ultimate license, create an HTTP streaming destination pointing at an unreachable host, for example https://localhost:9 (Secure > Audit events > Streams).

  2. Trigger an audit event in the group, or run in the Rails console:

    event = AuditEvent.where(entity_id: <group_id>).last
    AuditEvents::AuditEventStreamingWorker.perform_async('audit_operation', event.id)
  3. In log/exceptions_json.log the entry for the failed delivery now carries extra.error_category: "connection" alongside the destination fields.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Vasyl Pedak

Merge request reports

Loading
Loading