Skip to content

[Audit Events] Provide all audit events under a group via the API

Michael Becker requested to merge feat/337757-add-group-rollup-column into master

What does this MR do and why?

implements the query suggested by @ahegyi here

related to: #337757 (closed)

Query Plan

Query plan: https://explain.depesz.com/s/9INS

How to set up and validate locally

  1. Have an EE enabled GitLab instance
  2. find a grouo ID for a group you are owner of
  3. This group will need to have some project and group audit events. You can create some from the rails console using FactoryBot: 
    # using the admin user
user = User.find(1); nil
# flightjs/Flight project
project = Project.find(7); nil
# @flightjs group
group = project.group; nil 

# create audit events
FactoryBot.create(:project_audit_event, target_project: project, user: user); nil
2.times { FactoryBot.create(:group_audit_event, target_group: group, user: user) }; nil
  4. Using the group id, make a request to the API 
    curl -v \
     --header "PRIVATE-TOKEN: $TOKEN" \
     -H "Content-Type: application/json" \
     -H "Accept: application/json, text/plain, */*" \
     --url "localhost:3000/api/v4/groups/33/audit_events" | json_pp
  5. you should only see Group audit events in the response 
    [
 {
    "author_id" : 1,
    "created_at" : "2023-06-01T07:49:01.334Z",
    "details" : {
       "author_email" : "admin@example.com",
       "author_name" : "Jane Doe",
       "change" : "project_creation_level",
       "entity_path" : "flightjs",
       "from" : "",
       "ip_address" : "127.0.0.1",
       "target_details" : "Flightjs",
       "target_id" : 33,
       "target_type" : "Group",
       "to" : "Developers + Maintainers"
    },
    "entity_id" : 33,
    "entity_type" : "Group",
    "id" : 5
 },
 {
    "author_id" : 1,
    "created_at" : "2023-06-01T07:49:01.321Z",
    "details" : {
       "author_email" : "admin@example.com",
       "author_name" : "Jane Doe",
       "change" : "project_creation_level",
       "entity_path" : "flightjs",
       "from" : "",
       "ip_address" : "127.0.0.1",
       "target_details" : "Flightjs",
       "target_id" : 33,
       "target_type" : "Group",
       "to" : "Developers + Maintainers"
    },
    "entity_id" : 33,
    "entity_type" : "Group",
    "id" : 4
 }
]
  6. enabled the feature flag with the rails console 
    Feature.enable(:audit_event_group_rollup)
  7. make the same request, you should now see the project audit event in the response also 
    [
   {
      "author_id" : 1,
      "created_at" : "2023-06-01T07:49:01.334Z",
      "details" : {
         "author_email" : "admin@example.com",
         "author_name" : "Jane Doe",
         "change" : "project_creation_level",
         "entity_path" : "flightjs",
         "from" : "",
         "ip_address" : "127.0.0.1",
         "target_details" : "Flightjs",
         "target_id" : 33,
         "target_type" : "Group",
         "to" : "Developers + Maintainers"
      },
      "entity_id" : 33,
      "entity_type" : "Group",
      "id" : 5
   },
   {
      "author_id" : 1,
      "created_at" : "2023-06-01T07:49:01.321Z",
      "details" : {
         "author_email" : "admin@example.com",
         "author_name" : "Jane Doe",
         "change" : "project_creation_level",
         "entity_path" : "flightjs",
         "from" : "",
         "ip_address" : "127.0.0.1",
         "target_details" : "Flightjs",
         "target_id" : 33,
         "target_type" : "Group",
         "to" : "Developers + Maintainers"
      },
      "entity_id" : 33,
      "entity_type" : "Group",
      "id" : 4
   },
   {
      "author_id" : 1,
      "created_at" : "2023-06-01T07:48:45.375Z",
      "details" : {
         "author_email" : "admin@example.com",
         "author_name" : "Jane Doe",
         "change" : "packages_enabled",
         "entity_path" : "flightjs/Flight",
         "from" : "true",
         "ip_address" : "127.0.0.1",
         "target_details" : "Flight",
         "target_id" : 7,
         "target_type" : "Project",
         "to" : "false"
      },
      "entity_id" : 7,
      "entity_type" : "Project",
      "id" : 3
   }
]

Background Context

We want to be able to return all audit events under a given groups via a single endpoint. We have an issue, API endpoint to retrieve all audit events under a given group, to track this feature.

We actually had already implemented that feature, but the performance was so poor in production that it functionally did not work. The database query timed out almost 100% of the time.

The proposal was to improve the existing query's performance by using the InOperatorOptimization library.

Attempts to do this did not show a good performance boost (details here)

Another Approach

We then thought about implementing this feature with a new approach. The Idea is that we can add a new column, group_rollup, to the audit_events table. This column will be the id of the top-level group the audit event belongs to.

We could then update the Auditor to write this value at audit event creation, allowing us to implement the endpoint with a simple query:

SELECT *
FROM "audit_events"
WHERE "audit_events"."group_rollup_id" == @group.id

This query performed much better in database lab (details in this spike issue)

Back to InOperatorOptimization

In the MR for the group_rollup_id approach, it was discovered that with some tweaking and adding a new index, the InOperatorOptimization might just work

So this commit adds the application code for that approach.

The index additions are tracked in these issues:

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Michael Becker

Merge request reports