## Summary Group-level event type **denylist** filters for audit event streaming destinations are only evaluated at delivery time (inside `AuditEventStreamingWorker` / the NATS batched dispatcher). They are **not** considered by the enqueue pre-filter. As a result, every audit event of a denylisted type still creates a Sidekiq job (or NATS publish) that is picked up, evaluated, and then discarded without streaming anything. The **allowlist** filter does not have this problem. The enqueue pre-filter added in !229823 correctly skips enqueueing when no active destination allows the event type. When the denylist was introduced in !237996, the pre-filter was only adjusted to ignore `kind: deny` rows. We missed adding the equivalent denylist exclusion to the enqueue gate. Related: #633500. There, high-volume event types overwhelmed the streaming Sidekiq shard, and the denylist was considered as a mitigation but cannot reduce enqueue volume. ## Where the gap is Enqueue path: 1. `CommonAuditEventStreamable#stream_to_external_destinations` → `can_stream_to_external_destination?` 2. → `AuditEvents::ExternalDestinationStreamer#streamable?` (`streamers.any?(&:streamable?)`) 3. → `AuditEvents::Streaming::Group::Streamer#streamable?` → `external_audit_event_streaming_destinations.active.allowed_for_event_type(event_type).exists?` 4. `ExternallyStreamable.allowed_for_event_type` narrows filters to allowlist rows only: ```ruby # Restrict the pre-filter to allowlist entries when the filter model # supports a denylist. Denylist hits are caught by the per-event # allowed_to_stream? check at streaming time. matching_filters = matching_filters.allowlist if filter_class.respond_to?(:allowlist) where_not_exists(matching_filters) .or(where_exists(matching_filters.audit_event_type_in(event_type))) ``` A destination with **only** deny rows has no `kind: allow` rows, so `where_not_exists(matching_filters)` is true and the destination counts as streamable for every event type, including the denylisted ones. The denylist is only honored afterwards, in `ExternallyStreamable#event_type_allowed_to_stream?` → `event_type_denylisted?`, called from: - `AuditEvents::Streaming::BaseStreamer#execute` (Sidekiq path) - `AuditEvents::Streaming::BatchedDispatcher#dispatch_to_destination` (NATS path) ## Evidence from GitLab.com Group `9970` has destination `1000658` with only deny filters: | audit_event_type | kind | |---|---| | `repository_git_operation` | deny | | `user_authenticated_using_job_token` | deny | Rails console confirms the mismatch: ```ruby group = Group.find(9970) group.external_audit_event_streaming_destinations.active .allowed_for_event_type('repository_git_operation').pluck(:id) # => includes 1000658 (enqueue gate says "streamable") dest = AuditEvents::Group::ExternalStreamingDestination.find(1000658) dest.send(:event_type_allowed_to_stream?, 'repository_git_operation') # => false (worker drops it) ``` Kibana (`pubsub-sidekiq-inf-gprd*`, `json.class: "AuditEventStreamingWorker" and json.job_status: "done"`, filtered to `namespace_id: 9970` and `audit_event_type` in the two denylisted types) shows **~4.85M completed jobs in the last 24 hours**. That's about 56 jobs/s on average, peaking around 110 jobs/s, all of which do no delivery work. https://log.gprd.gitlab.net/app/r/s/zqkWr {width=900 height=113} ## Impact - Wasted Sidekiq capacity, Redis job storage, and DB queries (the worker re-runs `streamable?` and loads destinations) for events that will never be streamed. - The denylist cannot be used as a load-shedding lever for high-volume event types (for example `repository_git_operation`), which is exactly when customers or operators reach for it. - Behavior is inconsistent with the allowlist, which does prevent enqueueing. ## Proposal Exclude destinations whose denylist matches the event type in `allowed_for_event_type`, so the pre-filter matches the worker-side semantics: ``` destination is streamable for type T ⇔ NOT EXISTS(deny row for T) AND (NOT EXISTS(allow rows) OR EXISTS(allow row for T)) ``` Sketch: ```ruby allow_filters = destination_filters.allowlist deny_hits = destination_filters.denylist.audit_event_type_in(event_type) where_not_exists(allow_filters) .or(where_exists(allow_filters.audit_event_type_in(event_type))) .where_not_exists(deny_hits) ``` Because the gate is an `.exists?` over all active destinations, an event is still enqueued if **any** destination wants it. The worker-side `allowed_to_stream?` check stays in place for per-destination filtering. Notes: - No migration needed for group destinations, so this should be backportable. - Validate the query plan on postgres.ai. This runs on the hot path for every audit event. The existing uniqueness index on `(external_streaming_destination_id, audit_event_type)` should cover the new subquery. ### Specs to add - Destination with only a deny row matching the type → excluded. - Destination with only a deny row for a different type → included. - Destination with an allow row matching the type → included. - Mixed destinations on a group (one denies the type, one has no filters) → gate still returns true. ## Follow-up (out of scope here) Instance-level destinations (`AuditEvents::Instance::ExternalStreamingDestination`) do not support a denylist at all; only `event_type_filters` (allowlist) and `namespace_filters` exist. Parity would need a schema change on the instance filter table, so it should be tracked separately.