Backport of 'Support denylist filters for instance audit event streaming' to 19.5

Huzaifa Iftikharrequested to merge
635260-backport-instance-denylist-filters-19-5 into 19-5-stable-ee

What does this MR do and why?

Backports two merged MRs to 19-5-stable-ee so GitLab Dedicated can exclude high-volume event types (such as repository_git_operation) from instance audit event streaming destinations with a denylist filter:

  1. !261781 (merged) — Skip streaming enqueue for denylisted audit event types. Without this, denylisted events still get a Sidekiq job enqueued and are only dropped at send time.
  2. !261796 (merged) — Support denylist filters for instance audit event streaming. Adds a kind column to audit_events_instance_streaming_event_type_filters and wires the instance destination to denylist filters. Depends on the first MR.

This is the remediation for S1 incident INC-13995 (https://gitlab.com/gitlab-org/gitlab/-/work_items/633500). Git-over-HTTP traffic on a Dedicated tenant produced repository_git_operation audit events faster than the audit_events Sidekiq shard could process them. The backlog grew Redis memory to about 85% and forced operators to drop streaming jobs, so stream-only compliance events were permanently lost. Today the only option is an allowlist of every other event type, which silently stops streaming any event type added in a later release. Dedicated runs N-1 and is moving to 19.4.x next week (https://gitlab.com/gitlab-org/gitlab/-/work_items/633500#note_4054259366).

Backport details
  • Both commits were cherry-picked with git cherry-pick -x from the MRs' squash commits and applied without conflicts. Their diffs are identical to the merged MRs (patch IDs match).
  • The migration 20261013104217_add_kind_to_audit_events_instance_streaming_event_type_filters is a regular migration that adds one smallint DEFAULT 0 NOT NULL column to a small table, a metadata-only change in PostgreSQL. It keeps its original version, so instances that upgrade later to 19.6 or newer don't run it again.
  • The change is model-level only. Instance denylist filters are created through the Rails console, which Dedicated SREs can use. No GraphQL or UI changes are backported.
  • The group-level denylist support these changes rely on (the kind enum and allowlist/denylist scopes on AuditEvents::Group::EventTypeFilter) is already on this stable branch.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

  • This MR is backporting a bug fix, documentation update, or spec fix, previously merged in the default branch.
  • The MR that fixed the bug on the default branch has been deployed to GitLab.com (not applicable for documentation or spec changes).
  • The MR title is descriptive (e.g. "Backport of 'title of default branch MR'"). This is important, since the title will be copied to the patch blog post.
  • Required labels have been applied to this merge request
  • This MR has been approved by a maintainer (only one approval is required).
  • Ensure the E2E Omnibus GitLab EE pipeline (triggered by thee2e:test-on-omnibus-ee job) has succeeded. If it has failed (or shows warnings), investigate the failures. If you determine the failures are unrelated, you may proceed. If you need assistance investigating, request help in the #s_developer_experience Slack channel to confirm the failures are unrelated to the merge request.

Note to the merge request author and maintainer

If you have questions about the patch release process, please:

References

Edited by Huzaifa Iftikhar

Merge request reports