Bound S3 audit event streaming timeouts behind an ops flag
What does this MR do and why?
Caps audit event streaming sends to Amazon S3 at 8 seconds per connect and per read, with two retries, behind the audit_event_streaming_destination_timeouts ops flag (default off).
S3 destinations run on the AWS SDK defaults (15s connect, 60s read, 3 retries). When a send stalls, it holds a thread from the streaming worker's shared concurrency limit (1,200 jobs, shared by every customer) for a minute or more before the SDK retry succeeds. HTTP destinations are already capped at 8s by AuditEvents::HttpTimeoutConfig::DEFAULT, and this MR reuses those values.
Aws::S3Clientpasses extra options through toAws::S3::Client, so its defaults are unchanged for any caller that doesn't ask.AmazonS3StreamDestinationsets 8shttp_open_timeoutandhttp_read_timeoutwithretry_limit: 2(legacy retry mode) andmax_attempts: 3(standard and adaptive modes) when the flag is on.- The flag actor is the destination's top-level group, or
:instancefor instance destinations, so it can be enabled for one group first.
Timeouts surface as Seahorse::Client::NetworkingError, which !262597 (merged) already classifies as a timeout destination error, so they are logged and count towards the circuit breaker.
Production numbers (GitLab.com, last 24 hours unless noted)
S3 sends are fast, apart from occasional stalls that hit the SDK's 60s read timeout:
| Count | |
|---|---|
| Successful S3 sends | 6.36M, across 40 destinations |
| Failed S3 sends | 10.8k, all invalid credentials or wrong region |
| S3 sends that failed on a timeout | 0 |
For the group that accounts for almost all slow S3 jobs, job durations cluster exactly at the SDK timeout:
| Job duration | Jobs |
|---|---|
| under 8s | 196,658 (98.3%) |
| 8–55s | 139 |
| about 61s (one stall, then the retry succeeds) | 2,375 |
| about 122s (two stalls) | 1,086 |
| 150s and over | 0 |
- Stalls are bursty: in one 2-hour window, 246 of 469 stalled jobs fell within 6 minutes.
- The retry succeeds immediately, even mid-burst: jobs finish within about 1s of the 60s timeout.
- Thread cost: these stalls hold about 3 streaming threads on average, all day. Peak concurrency in the same period was 904 of 1,200.
Behaviour with the flag on
Measured with the SDK against a local socket that accepts connections and never responds:
| Today | Flag on | |
|---|---|---|
| Stall, then retry succeeds | about 61s | about 8.3s |
| Every attempt stalls | 60s per attempt, up to 4 attempts | fails after 25.0s (3 attempts) |
- Why two retries, not one: on the Sidekiq path the job does not retry a failed send, so running out of SDK retries loses the event for that destination. Because stalls are bursty, a second retry noticeably lowers that risk.
- Upper bound on lost events: the 1,086 jobs a day with two stalls only become losses if every attempt on the same destination stalls.
- Worst case per send: a connect that takes just under 8s followed by an 8s read stall gives about 50s in theory. Stalls observed in production are read stalls only.
- Write timeout: this SDK version has no write timeout option, so the body upload keeps Net::HTTP's default 60s write timeout. Payloads are small JSON objects.
Rollout and rollback
- Rollout: this is an
opsflag, so it has no rollout issue. Enable it for the group with the most slow S3 sends first, then watch S3 failures witherror_category: timeoutin Kibana (json.extra.destination_category: aws) before enabling it for the instance. - Rollback: disable the flag to return to the AWS SDK defaults.
References
- NATS audit event streaming epic: gitlab-org#17582
- Error classification for S3 timeouts: !262597 (merged)
Screenshots or screen recordings
Not applicable: no UI change.
How to set up and validate locally
-
Save this script, which points the SDK at a local socket that accepts connections and never replies, then run it with
bundle exec ruby stall.rb:require 'aws-sdk-s3' require 'socket' server = TCPServer.new('127.0.0.1', 0) Thread.new { loop { conn = server.accept; Thread.new { sleep 300; conn.close } } } client = Aws::S3::Client.new( region: 'us-east-1', endpoint: "http://127.0.0.1:#{server.addr[1]}", force_path_style: true, credentials: Aws::Credentials.new('a', 'b'), http_continue_timeout: nil, http_open_timeout: 8, http_read_timeout: 8, retry_limit: 2, max_attempts: 3 ) started = Process.clock_gettime(Process::CLOCK_MONOTONIC) begin client.put_object(bucket: 'b', key: 'k', body: '{}') rescue => e puts "#{e.class} after #{(Process.clock_gettime(Process::CLOCK_MONOTONIC) - started).round(1)}s" end -
It prints
Seahorse::Client::NetworkingError after 25.0s. Remove the four timeout and retry options to see the SDK defaults, which wait 60s per attempt. -
In a Rails console, with an S3 streaming destination configured for a top-level group, confirm the options are only applied when the flag is on:
destination = AuditEvents::Group::ExternalStreamingDestination.find_by(category: 'aws') stream_destination = AuditEvents::Streaming::Destinations::AmazonS3StreamDestination.for_batch(destination) stream_destination.send(:client_options) # => {} Feature.enable(:audit_event_streaming_destination_timeouts, destination.group) stream_destination.send(:client_options) # => {http_open_timeout: 8, http_read_timeout: 8, retry_limit: 2, max_attempts: 3}
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.