Skip to content

Allows filtering of streaming audit events

What does this MR do and why?

Implements API to allow streamed audit events to be filtered on a per-destination basis

Introduce APIs to:

  • List which events are being streamed to the given endpoint
query {
  group(fullPath: "compliance-tanuki") {
    id
    externalAuditEventDestinations {
      nodes {
        destinationUrl
        verificationToken
        id
        eventTypeFilters // collection of strings
      }
    }
  }
}
  • Cause the endpoint to receive a specific event
mutation createEventTypeFilter {
  auditEventsStreamingDestinationEventsCreate(input: {
    destinationId: "gid://gitlab/AuditEvents::ExternalAuditEventDestination/10",
    eventTypeFilters: ["repository_download_operation"]
  }){
    errors
    eventTypeFilters
  }
}
  • Cause the endpoint to no longer receive a specific event
mutation destroyEventTypeFilter {
  auditEventsStreamingDestinationEventsDestroy(input: {
    destinationId: "gid://gitlab/AuditEvents::ExternalAuditEventDestination/10",
    eventTypeFilters: ["repository_download_operation"]
  }){
    errors
    eventTypeFilters
  }
}

Migration logs on local

 rake db:migrate:up:main VERSION=20220911200106
main: == 20220911200106 CreateAuditEventsStreamingEventTypeFilters: migrating =======
main: -- create_table(:audit_events_streaming_event_type_filters, {})
main: -- quote_column_name(:audit_event_type)
main:    -> 0.0000s
main:    -> 0.0353s
main: == 20220911200106 CreateAuditEventsStreamingEventTypeFilters: migrated (0.0360s)

 rake db:migrate:down:main VERSION=20220911200106
main: == 20220911200106 CreateAuditEventsStreamingEventTypeFilters: reverting =======
main: -- drop_table(:audit_events_streaming_event_type_filters, {})
main:    -> 0.0024s
main: == 20220911200106 CreateAuditEventsStreamingEventTypeFilters: reverted (0.0039s)

Query Plans

rails query tested for 1000 records.

destination.event_type_filters.by_audit_event_type(event_type_filters).delete_all

query plan

EXPLAIN for: SELECT "audit_events_streaming_event_type_filters".* FROM "audit_events_streaming_event_type_filters" WHERE "audit_events_streaming_event_type_filters"."external_audit_event_destination_id" = 10 AND "audit_events_streaming_event_type_filters"."audit_event_type" IN ('4', '2', '3') 
                                                                           QUERY PLAN
----------------------------------------------------------------------------------------------------------------------------------------------------------------
 Index Scan using idx_event_type_filters_on_external_audit_event_destination_id on audit_events_streaming_event_type_filters  (cost=0.26..2.28 rows=1 width=34)
   Index Cond: (external_audit_event_destination_id = 10)
   Filter: (audit_event_type = ANY ('{4,2,3}'::text[]))

How to set up and validate locally

  1. Enable feature flag allow_audit_event_type_filtering
  2. Setup audit event streaming destination https://docs.gitlab.com/ee/administration/audit_event_streaming.html#add-a-new-event-streaming-destination
  3. Perform audit action. for example: download repository using zip download button.
  4. Check streaming is working.
  5. Create event type filter using auditEventsStreamingDestinationEventsCreate mutation use any event name other than audit action we are performing.
  6. Perform action again, audit event should be created but not streamed.
  7. Create event type filter using auditEventsStreamingDestinationEventsCreate mutation for the action we are performing. for example for zip download use eventTypeFilters: ["repository_download_operation"].
  8. Perform action again, audit event should be created but streamed.
  9. Remove this filter using auditEventsStreamingDestinationEventsDestroy mutation
  10. Perform action again, audit event should be created but not streamed.
  11. Remove all filters and perform any audit action, audit event should be created and streamed.

Numbered steps to set up and validate the change are strongly suggested.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #344845 (closed)

Edited by Harsimar Sandhu

Merge request reports