This epic is for streaming Audit Events. Current status: First iterations shipped and [documented](https://docs.gitlab.com/ee/administration/audit_event_streaming.html)! This epic contains issues and sub-epics for follow-on work. ## Table of Contents [[_TOC_]] [Video walkthrough](https://youtu.be/mjGtjX7IO60) of the epic. (current as of 2021-05-04) # Problem Statements and Use Cases Today, Audit Events are created inside GitLab and stored in our database. They can then be accessed and reviewed under the Audit Events tab for an Instance, Group, or Project. The results can be reviewed across our paginated viewer and individually examined to understand what is happening inside of GitLab. This is great visibility but there are some use cases that are difficult to address with this approach today. 1. _Use Case_: As a user, I want to be able to see, combine, manipulate, and report on all of the audit event data from my organization's multiple systems, including GitLab. 1. _Use Case_: As a user, I want to only look at specific audit events I care about so I can quickly answer the questions I am interested in. 1. _Use Case_: As a user, I want to have a full picture of what goes on inside GitLab and be able to review it after the fact. Our current approach also causes some issues for GitLab: 1. The audit events are stored in the same database as everything else so are subject to any performance issues in the whole database. 1. There are large, ongoing storage requirements for GitLab to store audit logs. 1. It will be difficult to effectively capture, manage, and store high-volume events, such as Git activity, in the current architecture. 1. GitLab data retention requirements may vary from our users' data retention requirements for audit logs. 1. New events can be difficult to add to GitLab since audit events are tightly coupled with many other parts of the GitLab application. Some of these problem statements are summarized visually in a board created during formation of this epic:  # Proposal Add a new audit event system that sends events as they occur to an external, user-defined destination. Allow users to customize which events are sent or not, so they can manage what they receive. Notably, the existing audit event system should remain in place. Webhook-based audit events are complementary and additive to the existing system, rather than a replacement of the existing system.  ## "Classic" Audit Events The existing audit events, "Classic", system should remain in place. The goal of classic audit events will be to maintain a single-source-of-truth of the most criticial audit events that could effect a GitLab instance, directly inside GitLab. For more concrete examples, the first four event types on our [prioritization list](https://about.gitlab.com/direction/manage/compliance/audit-events/#how-we-will-prioritize-adding-new-audit-events) would likely remain as classic audit events within GitLab. There may be times when an audit event is included in both classic audit events as well as webhook-based audit events. This is a good thing, as those critical events in classic audit events will likely also be of interest if they are streamed to an external endpoint. ## Licensing ~"GitLab Ultimate" ## Iteration Plan (update as needed) This will be a very deep capability, so we need to plan on finding small iterations that get to the desired end state gradually. Several proposed iterations are below. Each of the below iterations should be broken down into individual issues or epics to capture each iteration as a "vertical slice". <details><summary>:white_check_mark: Completed! - Internal-only, "tech preview" </summary> ### Iteration 0 - Internal-only, "tech preview" - :white_check_mark: Completed!  An initial iteration to explore the concept would focus on: 1. Add ability to select a subset of different audit events to send to webhooks. 1. Add this under the existing Compliance tab or add a new configuration tab at the root group level. 1. Should be able to manage all project-level events at this screen. 1. For self-hosted, instance level events add a separate screen to handle instance-specific events. 1. Send audit events to a temporary GitLab system that is not the current database. 1. Identifying any technical or UX challenges we'd need to solve before exposing to users. #### Success criteria 1. Basic technical implementation to show what is possible. 1. GraphQL API only, no user interface 1. User-facing demo/slideshow/story to gather feedback. 1. Delivered in either a recorded video or sync meeting, rather than self-service in an all repos. 1. User feedback received and recorded on what is good/bad/missing. 1. Feedback used to create new or update existing epics and issues. </details> <details> <summary>:white_check_mark: Completed! - User-facing, Minimal and Viable </summary> ### Iteration 1 - User-facing, Minimal and Viable  This iteration would be the first user-facing delivery. It would build on the first iteration and add: 1. The ability to specify a destination for the audit events as they occur. 1. Add this in the same place as event subsets are chosen NOTE: Filtering by event type was not done in this iteration and will be done in a future set of issues </details> ### Iteration 2 - Event filtering and grouping by type  Child epic: https://gitlab.com/groups/gitlab-org/-/epics/6980 This iteration will be about allowing customization of the type of events that are sent to a given endpoint. Currently _all_ audit events are sent to a given destination, which may not be desirable for the user due to the data volume and the need for them to filter and process the data. At a minimum we should let users select each individual audit event type to be sent. We should also offer the ability to do "grouping" of audit events based on type. For example, rather than requiring the user to select 5 different checkboxes related to MR approval settings, we should offer one checkbox that encapsulates all 5 of those events. See the concept diagram above for how this might work. ### Iteration 3 - Additional API-endpoint and storage types supported  This iteration would focus on giving more flexibility on the specific destination type that events were sent. It would focus on: 1. Add options to select pre-determined destination types from: 1. Amazon [S3 buckets](https://aws.amazon.com/s3/) 1. Google [Cloud Storage](https://cloud.google.com/storage) 1. Splunk 1. Update message format to be compatible with the chosen destination type # Considerations # Open Questions 1. When we say "streaming" what is meant by that? 1. Do users need true real time streaming, near-real time streaming, or batch streaming. 1. _Intuition and working assumption is that near-real time is fine, but need to validate this._ 1. _We would expect this to be a webhook-type service, not a "stream" in the C++ object sense that has retries, failure handling logic_ 1. What format and technology do we want to use? 1. Security requirements? 1. Do we require any sort of authentication before sending? Encrypted/cleartext? Dig in as part of implementation planning. 1. Do they need requests to be signed by GitLab (or the self-hosted instance yes)? 1. _Intuition says yes_ 1. When users change the reported audit event subset, what expectations should be set? 1. Are older events lost, have they been bundled? 1. _Intuition is that events are not stored in between enabled/disabled state_ 1. Who has permission to change these settings? 1. What is the relationship between instance, group, and project level audit events? 1. _Intuition is that we should find a single-source-of-truth to blend these together, rather than have multiple screens. In the case where SaaS and SM must be different, keep groups and project together and only split out instances._ 1. _Add to this list organically_ *This page may contain information related to upcoming products, features and functionality. It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes. Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc.*