Unhandled exception streaming audit events to AWS S3

Summary

AWS S3 audit log streaming is failing on GitLab.com since August 20th. We can see an increase in related exceptions since that time:

  "class": "NoMethodError",
        "message": "undefined method `downcase' for nil:NilClass",
      "exception": {
        "backtrace": [
          "ee/lib/audit_events/streaming/destinations/amazon_s3_stream_destination.rb:39:in `filename'",
          "ee/lib/audit_events/streaming/destinations/amazon_s3_stream_destination.rb:9:in `stream'",
          "ee/lib/audit_events/streaming/base_streamer.rb:57:in `stream_to_destination'",
          "ee/lib/audit_events/streaming/base_streamer.rb:46:in `track_and_stream'",
          "ee/lib/audit_events/streaming/base_streamer.rb:34:in `block in execute'",

We've also had customer's reach out to GitLab support with this problem.

Internal Slack thread regarding the discussion.

Steps to reproduce

I've been able to reproduce this problem on GitLab.com

  1. Setup AWS S3 audit event streaming for a top-level namespace on GitLab.com
  2. Generate group audit event logs
  3. Check the Sidekiq log for meta caller id AuditEvents::AuditEventStreamingWorker and exception class NoMethodError

Example Project

What is the current bug behavior?

Audit event streaming fails during the generating of the filename, as it relies on the audit_event['event_type'], which is nil.

What is the expected correct behavior?

Audit event streaming succeeds and data is sent to AWS S3 bucket

Relevant logs and/or screenshots

Search query for Sidekiq indices:

json.meta.caller_id:"AuditEvents::AuditEventStreamingWorker" AND json.exception.class:"NoMethodError"

https://log.gprd.gitlab.net/app/r/s/kqAkX

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info 

 (For installations with omnibus-gitlab package run and paste the output of: \`sudo gitlab-rake gitlab:env:info\`)  (For installations from source run and paste the output of: \`sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production\`)

Results of GitLab application Check

Expand for output related to the GitLab application check 

(For installations with omnibus-gitlab package run and paste the output of: `sudo gitlab-rake gitlab:check SANITIZE=true`)

(For installations from source run and paste the output of: `sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true`)

(we will only investigate if the tests are passing)

Possible fixes

Patch release information for backports

If the bug fix needs to be backported in a patch release to a version under the maintenance policy, please follow the steps on the patch release runbook for GitLab engineers.

Refer to the internal "Release Information" dashboard for information about the next patch release, including the targeted versions, expected release date, and current status.

High-severity bug remediation

To remediate high-severity issues requiring an internal release for single-tenant SaaS instances, refer to the internal release process for engineers.

Edited by 🤖 GitLab Bot 🤖