Let each streaming destination classify its own delivery errors
What does this MR do and why?
Changes the structure of audit event streaming delivery so that delivery outcome tracking can be added on top of it. No metrics are added here; the follow-up MRs are listed under References.
The main decision is to encapsulate error classification within each stream destination. Until now a single ErrorClassifier module held every destination's SDK exceptions in one list (AWS service and networking errors, GCP auth and key errors, the HTTP client's transport errors) and answered one boolean, log or Sentry. Adding a destination meant editing that shared list, and both delivery paths re-checked the exception to make the same decision. Azure Blob, Splunk, Datadog and Snowflake destinations are all in the backlog, so this would keep growing.
After this MR:
BaseStreamDestinationowns the fixed list of error categories (timeout,connection,auth,not_found,rate_limited,client_error,server_error,invalid_config,internal_error) and the transport mapping that every destination shares, since HTTP and GCP both useGitlab::HTTPand S3 errors unwrap to the same socket exceptions.AmazonS3StreamDestination#classify_errormaps the AWS SDK: service errors by HTTP status,SlowDown(the only throttling code S3 returns) torate_limited,PermanentRedirect(bucket in another region) toinvalid_config, networking errors by their wrapped cause, missing credentials, invalid region. Then falls back to the shared mapping.GoogleCloudLoggingStreamDestination#classify_errormaps Signet authorisation and RSA key errors and falls back to the shared mapping. These two cover everything the GCP client stack raises for a bad configuration: googleauth retries every token fetch failure and wraps it inSignet::AuthorizationError, and a malformed PEM raisesOpenSSL::PKey::RSAError. Anything else from the GCP adapter is a GitLab bug and now reaches Sentry, where master's catch-all logged it as customer configuration.HttpStreamDestinationuses the shared mapping unchanged.BaseStreamDestination#deliverand#deliver_batchwrap the subclass'sstreamcalls and never raise. They return anOutcomewith the error and its category, and call a private no-oprecord_delivery(outcome, count)hook where the metrics will plug in. The hook call is wrapped in a rescue, so a bug in it is reported to Sentry without changing the outcome. A category outsideERROR_CATEGORIESis treated asinternal_error, so a subclass bug cannot pass a GitLab error off as a customer one.BaseStreamer(Sidekiq path) andBatchedDispatcher(NATS path) branch on the outcome (outcome.internal_error?) instead of rescuing and classifying themselves. Classification happens once, in the destination. Their remaining rescue only covers errors raised outside the delivery call, such as internal event tracking, an unknown category, or the success and failure handlers themselves, which are GitLab-side by definition.ErrorClassifierand thelog_only_errorshook are removed. The category-to-adapter registry becomesDestinations.for_category, next to the adapters, and both callers use it instead of reaching into a sibling's constant.- Local variables and spec doubles that hold adapter instances are named
stream_destinationrather thanstreamer, so they are not confused with the realBaseStreamer.
Logging and error tracking cleanup that rides along:
- Failure log context now carries
error_category, so Kibana can break failures down by cause without confusing it withexception.class.
Routing changes worth a reviewer's eye, all a consequence of classifying per destination:
Seahorse::Client::NetworkingError(S3 unreachable) andAws::Errors::MissingCredentialsErrorpreviously fell through to Sentry with no circuit breaker count. They are now destination-side (connectionandauth), so they are logged and count toward the breaker.- Non-S3
Aws::Errors::ServiceErrorsubclasses are now destination-side too. Master only matched the S3 subclass, which is the only one the code can raise today. - For GCP destinations, master's
log_only_errors = [StandardError]made every error log-only, including GitLab bugs inside the adapter (aNoMethodErrorin payload building, an oversized event hitting the JSON size limit) and errors raised around the delivery call (internal event tracking, an unknown category, building the batch). All of those are nowinternal_error: Sentry, andfailedon the NATS path so the batch is retried. Only Signet and RSA key errors stay destination-side for GCP. OpenSSL::PKey::RSAErrorandSignet::AuthorizationErrorare only classified by the GCP destination. An HTTP or S3 destination raising them, which cannot happen with the current clients, would now beinternal_error.
References
- Implementation issue: https://gitlab.com/gitlab-org/gitlab/-/work_items/633532
- Investigation and approved plan: #577415 (closed)
Stacked follow-ups
These Draft MRs target this branch and retarget to master when it merges. The metrics MRs plug into the record_delivery hook and are stacked on each other in this order:
- !262609 (merged) adds
gitlab_audit_event_streaming_delivery_events_totalwithcategory,outcome(success,internal_error,destination_error) anderror_categorylabels. It replaces the first attempt in !261765 (closed). - !262610 (merged) adds
gitlab_audit_event_streaming_delivery_duration_secondsper category, with buckets up to 300s so the current S3 and GCP tail can be baselined before their timeouts are cut to 8s (https://gitlab.com/gitlab-org/gitlab/-/work_items/636583). - !262612 (merged) adds
gitlab_audit_event_streaming_circuit_breaker_skipped_totalfor attempts skipped while a destination's circuit is open. - !266796 (merged) logs an
infoline for sends that take 4s or more or get a non-2xx response, with destination, root namespace, duration and status, and moves the per-event "Audit event streamed" line to debug. - !266797 (closed) logs one
audit_streaming_deliveriessummary field per job on the streaming worker and the NATS consumer.
Beside the metrics stack:
- !266404 (merged) makes a non-2xx response from an HTTP or GCP destination a classified failure. Today both adapters return the response without checking the status, so a rejected write counts as delivered; this MR does not change that, the follow-up does.
Screenshots or screen recordings
Not applicable, no UI changes.
How to set up and validate locally
-
In a group with an Ultimate license, create an HTTP streaming destination pointing at an unreachable host, for example
https://localhost:9(Secure > Audit events > Streams). -
Trigger an audit event in the group, or run in the Rails console:
event = AuditEvent.where(entity_id: <group_id>).last AuditEvents::AuditEventStreamingWorker.perform_async('audit_operation', event.id) -
In
log/exceptions_json.logthe entry for the failed delivery now carriesextra.error_category: "connection"alongside the destination fields.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.