Treat a non-2xx response as a failed audit event delivery
What does this MR do and why?
The HTTP and Google Cloud Logging audit event streaming destinations return the response from Gitlab::HTTP.post without looking at the status code. Gitlab::HTTP only raises for transport failures, so a 401, 403, 415 or 500 from the destination comes back as a plain response and is counted as a successful delivery: the circuit breaker records a success, the "Audit event streamed" log line carries the status as if it were fine, and on the NATS path the batch is acked and the events are gone.
This MR adds AuditEvents::Streaming::Destinations::DestinationError, which carries the HTTP status and nothing else (the response body is customer data and the message ends up in logs and Sentry). The HTTP and GCP destinations raise it for any non-2xx response, and BaseStreamDestination#classify_error maps it through the same status categories the S3 destination already uses: 401 and 403 to auth, 404 to not_found, 429 to rate_limited, other 4xx to client_error, 5xx to server_error.
This is a behaviour change, so it is behind the gitlab_com_derisk feature flag audit_event_streaming_fail_on_non_2xx_response (rollout issue #638681), disabled by default and checked per top-level group (:instance for instance destinations). With the flag on, a 4xx or 5xx response trips the circuit breaker like a transport error does, and on NATS the batch is left unacked and redelivered the same way a timeout is. Once the breaker opens, the destination is skipped for the open window and batches that have no other destination are acked and dropped, exactly as happens for transport failures today. A destination that keeps answering 429 is backed off the same way; Retry-After is not read.
Production evidence
An earlier version of this description claimed there were no 4xx or 5xx responses in 15 days of logs. That search used the wrong field: BaseStreamer#log_streaming_success logs the status under Labkit::Fields::HTTP_STATUS_CODE, which is json.status in Kibana, not json.http_status_code.
Searching json.status on pubsub-sidekiq-inf-gprd* over a recent 24-hour window (analysis by @huzaifaiftikhar1, figures in an internal note on this MR) shows that a few percent of all "Audit event streamed" lines are non-2xx, spread over dozens of destinations and almost all HTTP. 401, 415 and 403 lead, followed by 530, 400, 500 and 503. A dozen destinations reject every send, and a couple more reject a quarter to a third of theirs and never trip the breaker because their successes reset the count. All of these are recorded as successful deliveries today.
Expected effect with the flag on
- Sidekiq path: destinations that reject everything trip the breaker (25 failures in 5 minutes) and are skipped for 5 minutes, so they are held to roughly 25 sends per cycle. Partly failing destinations still never trip, because a success resets the count; that is a separate follow-up.
- Logs: the non-2xx share of the "Audit event streamed" info lines becomes failure logs in
exceptions_json; the breaker cuts most of them. - NATS path: 5xx, 408 and 429 batches are retried, which is right. Permanent 4xx (400, 401, 403, 404, 415) classify as
auth,not_foundorclient_error, which !268096 (merged) treats as permanent failures on the NATS path: withaudit_event_streaming_nats_skip_permanent_failure_redeliveryon, the batch is acked and the failure recorded instead of being redelivered to the group's other destinations. With that flag off they are redelivered until the stream's 24hmax_age(10 deliveries once !267213 (merged) lands). Today NATS is enabled for one top-level group.
The user documentation sentence about retries on error status is left for the MR that enables the flag by default.
References
- https://gitlab.com/gitlab-org/gitlab/-/work_items/633532
- !262597 (merged) (destination error classification, merged)
Screenshots or screen recordings
Not applicable.
How to set up and validate locally
- Make sure the instance has an Ultimate license and that
external_audit_eventsis available. - Create an HTTP streaming destination for a group (or the instance) pointing at
https://httpbin.org/status/403. - Trigger an audit event in that group, for example by changing the group description. With the flag off the delivery is logged as a success in
log/application_json.logwith"status": 403. - Enable the flag for the group in a Rails console:
Feature.enable(:audit_event_streaming_fail_on_non_2xx_response, Group.find_by_full_path('<group>'))(or:instancefor an instance destination). - Trigger another event. The delivery is now logged in
log/exceptions_json.logasAuditEvents::Streaming::Destinations::DestinationErrorwith"message": "Destination responded with HTTP 403"and"error_category": "auth". - Point the destination at
https://httpbin.org/status/200and trigger another event: nothing is logged inexceptions_json.logand the delivery succeeds.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.