Bound S3 audit event streaming timeouts behind an ops flag

What does this MR do and why?

Caps audit event streaming sends to Amazon S3 at 8 seconds per connect and per read, with two retries, behind the audit_event_streaming_destination_timeouts ops flag (default off).

S3 destinations run on the AWS SDK defaults (15s connect, 60s read, 3 retries). When a send stalls, it holds a thread from the streaming worker's shared concurrency limit (1,200 jobs, shared by every customer) for a minute or more before the SDK retry succeeds. HTTP destinations are already capped at 8s by AuditEvents::HttpTimeoutConfig::DEFAULT, and this MR reuses those values.

  • Aws::S3Client passes extra options through to Aws::S3::Client, so its defaults are unchanged for any caller that doesn't ask.
  • AmazonS3StreamDestination sets 8s http_open_timeout and http_read_timeout with retry_limit: 2 (legacy retry mode) and max_attempts: 3 (standard and adaptive modes) when the flag is on.
  • The flag actor is the destination's top-level group, or :instance for instance destinations, so it can be enabled for one group first.

Timeouts surface as Seahorse::Client::NetworkingError, which !262597 (merged) already classifies as a timeout destination error, so they are logged and count towards the circuit breaker.

Production numbers (GitLab.com, last 24 hours unless noted)

S3 sends are fast, apart from occasional stalls that hit the SDK's 60s read timeout:

Count
Successful S3 sends 6.36M, across 40 destinations
Failed S3 sends 10.8k, all invalid credentials or wrong region
S3 sends that failed on a timeout 0

For the group that accounts for almost all slow S3 jobs, job durations cluster exactly at the SDK timeout:

Job duration Jobs
under 8s 196,658 (98.3%)
8–55s 139
about 61s (one stall, then the retry succeeds) 2,375
about 122s (two stalls) 1,086
150s and over 0
  • Stalls are bursty: in one 2-hour window, 246 of 469 stalled jobs fell within 6 minutes.
  • The retry succeeds immediately, even mid-burst: jobs finish within about 1s of the 60s timeout.
  • Thread cost: these stalls hold about 3 streaming threads on average, all day. Peak concurrency in the same period was 904 of 1,200.
Behaviour with the flag on

Measured with the SDK against a local socket that accepts connections and never responds:

Today Flag on
Stall, then retry succeeds about 61s about 8.3s
Every attempt stalls 60s per attempt, up to 4 attempts fails after 25.0s (3 attempts)
  • Why two retries, not one: on the Sidekiq path the job does not retry a failed send, so running out of SDK retries loses the event for that destination. Because stalls are bursty, a second retry noticeably lowers that risk.
  • Upper bound on lost events: the 1,086 jobs a day with two stalls only become losses if every attempt on the same destination stalls.
  • Worst case per send: a connect that takes just under 8s followed by an 8s read stall gives about 50s in theory. Stalls observed in production are read stalls only.
  • Write timeout: this SDK version has no write timeout option, so the body upload keeps Net::HTTP's default 60s write timeout. Payloads are small JSON objects.
Rollout and rollback
  • Rollout: this is an ops flag, so it has no rollout issue. Enable it for the group with the most slow S3 sends first, then watch S3 failures with error_category: timeout in Kibana (json.extra.destination_category: aws) before enabling it for the instance.
  • Rollback: disable the flag to return to the AWS SDK defaults.

References

Screenshots or screen recordings

Not applicable: no UI change.

How to set up and validate locally

  1. Save this script, which points the SDK at a local socket that accepts connections and never replies, then run it with bundle exec ruby stall.rb:

    require 'aws-sdk-s3'
    require 'socket'
    
    server = TCPServer.new('127.0.0.1', 0)
    Thread.new { loop { conn = server.accept; Thread.new { sleep 300; conn.close } } }
    
    client = Aws::S3::Client.new(
      region: 'us-east-1', endpoint: "http://127.0.0.1:#{server.addr[1]}", force_path_style: true,
      credentials: Aws::Credentials.new('a', 'b'), http_continue_timeout: nil,
      http_open_timeout: 8, http_read_timeout: 8, retry_limit: 2, max_attempts: 3
    )
    
    started = Process.clock_gettime(Process::CLOCK_MONOTONIC)
    begin
      client.put_object(bucket: 'b', key: 'k', body: '{}')
    rescue => e
      puts "#{e.class} after #{(Process.clock_gettime(Process::CLOCK_MONOTONIC) - started).round(1)}s"
    end
  2. It prints Seahorse::Client::NetworkingError after 25.0s. Remove the four timeout and retry options to see the SDK defaults, which wait 60s per attempt.

  3. In a Rails console, with an S3 streaming destination configured for a top-level group, confirm the options are only applied when the flag is on:

    destination = AuditEvents::Group::ExternalStreamingDestination.find_by(category: 'aws')
    stream_destination = AuditEvents::Streaming::Destinations::AmazonS3StreamDestination.for_batch(destination)
    
    stream_destination.send(:client_options) # => {}
    Feature.enable(:audit_event_streaming_destination_timeouts, destination.group)
    stream_destination.send(:client_options) # => {http_open_timeout: 8, http_read_timeout: 8, retry_limit: 2, max_attempts: 3}

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Merge request reports

Loading
Loading