Filter streaming audit events by namespace at group level
### Release Notes
Streaming audit events have been extended to support filtering by sub-group or project at the group level, in addition to the existing support for event type filtering.
This additional filter will allow you to separate out events in your streams to send to different destinations, or to exclude irrelevant sub-groups/projects, ensuring you have the most actionable events for your team to monitor.
### Proposal
Currently Streaming audit events can be filtered by event type https://docs.gitlab.com/ee/administration/audit_event_streaming.html#event-type-filters
Similar to event type it would be great to be able to also filter by sub group/project. The use case is so users can filter out events from a specific subgroup or project to seperate destination for automation and monitoring.
Use case:
- Send all audit events from the group/subgroup1 and its projects to the destination1.
- Send all audit events from the group/subgroup2 and its projects to the destination.
### Implementation Plan
### Requirements
1. Filters for subgroup or project events belonging to a group to be introduced for group level external http audit event destinations.
2. Currently there should be only 1 filter for filtering a subgroup or a project related events. Either a project can be added for filter or a subgroup. This limit can be removed in future or increased.
3. These filters will work along with the existing event type filters.
4. If there is a filter added for a subgroup then events belonging to it, its projects, its subgroups, subgroups' subgroups, subgroups' projects will be streamed.
5. Projects and subgroups can be any descendant of the group not only the direct children.
### Database schema
1. Create a table for storing the information related to the filter added:
1. audit_events_streaming_http_group_namespace_filters
1. Attributes:
1. external_audit_event_destination_id
2. namespace_id (group_id or project namespace id)
2. Deletion of row if the corresponding destination or namespace gets deleted.
3. Composite unique index on external_audit_event_destination_id and namespace_id columns.
### Models
1. Need to create a model corresponding to the above table with relationship with the [`ExternalAuditEventDestination`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/models/audit_events/external_audit_event_destination.rb) as well as namespace model.
2. In [`ExternalAuditEventDestination`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/models/audit_events/external_audit_event_destination.rb) we would have to add validations for ensuring that no more than 1 subgroup or project filter is added to the destination.
### Business logic
1. While streaming audit events to a certain group level http destination, we need to add a check whether there is a subgroup or project filter present for the destination and if it exists then check if the audit event entity matches with the subgroup or project for which the filter is present. This can be added in method [`allowed_to_stream?`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/models/concerns/audit_events/custom_http_externally_destinationable.rb#L28).
2. Filtering:
1. For projects filter, the filters are going to be quite straightforward with streaming all the events with `entity` matching the filtered project will be streamed to the destination.
2. For groups/subgroups filter, we need to check whether the group/subgroup id is present in the list of ids of self or parent groups. For audit event with entity type group, it would be returned by `self_and_ancestor_ids` method and for events with entity type project, it would be returned by `self_and_ancestor_ids` method for the parent of project.
### GraphQL Mutations and Queries
Following changes in current mutations and queries will be required
#### List all the subgroup and project filters for a destination
```graphql
query {
group(fullPath: "my-group") {
id
externalAuditEventDestinations {
nodes {
destinationUrl
verificationToken
id
name
headers {
nodes {
key
value
id
}
}
eventTypeFilters
namespaceFilter {
id
namespace {
id
name
fullPath
}
}
}
}
}
}
```
#### Create subgroup filters for destination
```graphql
mutation {
auditEventsStreamingHTTPNamespaceFiltersAdd(
input: {destinationId: "gid://gitlab/AuditEvents::AuditEventDestination/341", namespaceId: "gid://gitlab/Group/22"}
) {
errors
namespaceFilter {
nodes {
id
namespace {
id
name
fullPath
}
}
}
}
}
```
#### Remove subgroup filters for destination
```graphql
mutation auditEventsStreamingHttpNamespaceFiltersDelete {
auditEventsStreamingHttpNamespaceFiltersDestroy(input: {
namespaceFilterId: "gid://gitlab/AuditEvents::Streaming::HTTP::NamespaceFilter/5"
}) {
errors
}
}
```
epic