Skip to content

Add event streaming metrics

Max Woolf requested to merge 344670-audit-streaming-usage-data into master

What does this MR do and why?

Adds two new service ping metrics:

  • Count of all audit streaming destinations
  • Distinct count of all groups that have at least 1 audit streaming destination added.

Query Review

All Streaming Destinations

SELECT MIN("audit_events_external_audit_event_destinations"."id") FROM "audit_events_external_audit_event_destinations" WHERE "audit_events_external_audit_event_destinations"."created_at" BETWEEN '2021-10-31 08:59:51.800098' AND '2021-11-28 08:59:51.800253' /*application:console,db_config_name:main,line:/lib/gitlab/database/batch_counter.rb:127:in `actual_start'*/

 Result  (cost=3.15..3.16 rows=1 width=8) (actual time=5.995..5.996 rows=1 loops=1)
   Buffers: shared read=2
   I/O Timings: read=5.965 write=0.000
   InitPlan 1 (returns $0)
     ->  Limit  (cost=0.12..3.15 rows=1 width=8) (actual time=5.989..5.990 rows=1 loops=1)
           Buffers: shared read=2
           I/O Timings: read=5.965 write=0.000
           ->  Index Scan using audit_events_external_audit_event_destinations_pkey on public.audit_events_external_audit_event_destinations  (cost=0.12..3.15 rows=1 width=8) (actual time=5.987..5.987 rows=1 loops=1)
                 Index Cond: (audit_events_external_audit_event_destinations.id IS NOT NULL)
                 Filter: ((audit_events_external_audit_event_destinations.created_at >= '2021-10-31 08:59:51.800098+00'::timestamp with time zone) AND (audit_events_external_audit_event_destinations.created_at <= '2021-11-28 08:59:51.800253+00'::timestamp with time zone))
                 Rows Removed by Filter: 0
                 Buffers: shared read=2
                 I/O Timings: read=5.965 write=0.000


Time: 6.686 ms  
  - planning: 0.634 ms  
  - execution: 6.052 ms  
    - I/O read: 5.965 ms  
    - I/O write: 0.000 ms  
  
Shared buffers:  
  - hits: 0 from the buffer pool  
  - reads: 2 (~16.00 KiB) from the OS file cache, including disk I/O  
  - dirtied: 0  
  - writes: 0  
SELECT MAX("audit_events_external_audit_event_destinations"."id") FROM "audit_events_external_audit_event_destinations" WHERE "audit_events_external_audit_event_destinations"."created_at" BETWEEN '2021-10-31 08:59:51.800098' AND '2021-11-28 08:59:51.800253'

 Result  (cost=3.15..3.16 rows=1 width=8) (actual time=0.028..0.029 rows=1 loops=1)
   Buffers: shared hit=2
   I/O Timings: read=0.000 write=0.000
   InitPlan 1 (returns $0)
     ->  Limit  (cost=0.12..3.15 rows=1 width=8) (actual time=0.024..0.024 rows=1 loops=1)
           Buffers: shared hit=2
           I/O Timings: read=0.000 write=0.000
           ->  Index Scan using audit_events_external_audit_event_destinations_pkey on public.audit_events_external_audit_event_destinations  (cost=0.12..3.15 rows=1 width=8) (actual time=0.022..0.022 rows=1 loops=1)
                 Index Cond: (audit_events_external_audit_event_destinations.id IS NOT NULL)
                 Filter: ((audit_events_external_audit_event_destinations.created_at >= '2021-10-31 08:59:51.800098+00'::timestamp with time zone) AND (audit_events_external_audit_event_destinations.created_at <= '2021-11-28 08:59:51.800253+00'::timestamp with time zone))
                 Rows Removed by Filter: 0
                 Buffers: shared hit=2
                 I/O Timings: read=0.000 write=0.000

Time: 0.216 ms  
  - planning: 0.153 ms  
  - execution: 0.063 ms  
    - I/O read: 0.000 ms  
    - I/O write: 0.000 ms  
  
Shared buffers:  
  - hits: 2 (~16.00 KiB) from the buffer pool  
  - reads: 0 from the OS file cache, including disk I/O  
  - dirtied: 0  
  - writes: 0  
SELECT COUNT("audit_events_external_audit_event_destinations"."id") FROM "audit_events_external_audit_event_destinations" WHERE "audit_events_external_audit_event_destinations"."created_at" BETWEEN '2021-10-31 08:59:51.800098' AND '2021-11-28 08:59:51.800253' AND "audit_events_external_audit_event_destinations"."id" >= 5 AND "audit_events_external_audit_event_destinations"."id" < 9

 Aggregate  (cost=3.15..3.16 rows=1 width=8) (actual time=0.066..0.067 rows=1 loops=1)
   Buffers: shared hit=4
   I/O Timings: read=0.000 write=0.000
   ->  Index Scan using audit_events_external_audit_event_destinations_pkey on public.audit_events_external_audit_event_destinations  (cost=0.12..3.15 rows=1 width=8) (actual time=0.018..0.019 rows=0 loops=1)
         Index Cond: ((audit_events_external_audit_event_destinations.id >= 5) AND (audit_events_external_audit_event_destinations.id < 9))
         Filter: ((audit_events_external_audit_event_destinations.created_at >= '2021-10-31 08:59:51.800098+00'::timestamp with time zone) AND (audit_events_external_audit_event_destinations.created_at <= '2021-11-28 08:59:51.800253+00'::timestamp with time zone))
         Rows Removed by Filter: 0
         Buffers: shared hit=4
         I/O Timings: read=0.000 write=0.000

Time: 0.299 ms  
  - planning: 0.164 ms  
  - execution: 0.135 ms  
    - I/O read: 0.000 ms  
    - I/O write: 0.000 ms  
  
Shared buffers:  
  - hits: 4 (~32.00 KiB) from the buffer pool  
  - reads: 0 from the OS file cache, including disk I/O  
  - dirtied: 0  
  - writes: 0

Distinct Group Count

SELECT MIN("namespaces"."id") FROM "namespaces" INNER JOIN "audit_events_external_audit_event_destinations" ON "audit_events_external_audit_event_destinations"."namespace_id" = "namespaces"."id" WHERE "namespaces"."type" = 'Group'

 Aggregate  (cost=5.10..5.11 rows=1 width=4) (actual time=11.133..11.136 rows=1 loops=1)
   Buffers: shared hit=3 read=6
   I/O Timings: read=10.918 write=0.000
   ->  Nested Loop  (cost=0.56..5.10 rows=1 width=4) (actual time=11.118..11.125 rows=1 loops=1)
         Buffers: shared hit=3 read=6
         I/O Timings: read=10.918 write=0.000
         ->  Index Only Scan using index_external_audit_event_destinations_on_namespace_id on public.audit_events_external_audit_event_destinations  (cost=0.12..1.64 rows=1 width=8) (actual time=3.332..3.336 rows=1 loops=1)
               Heap Fetches: 0
               Buffers: shared read=2
               I/O Timings: read=3.245 write=0.000
         ->  Index Scan using namespaces_pkey on public.namespaces  (cost=0.43..3.46 rows=1 width=4) (actual time=7.777..7.777 rows=1 loops=1)
               Index Cond: (namespaces.id = audit_events_external_audit_event_destinations.namespace_id)
               Filter: ((namespaces.type)::text = 'Group'::text)
               Rows Removed by Filter: 0
               Buffers: shared hit=3 read=4
               I/O Timings: read=7.673 write=0.000

Time: 13.783 ms  
  - planning: 2.593 ms  
  - execution: 11.190 ms  
    - I/O read: 10.918 ms  
    - I/O write: 0.000 ms  
  
Shared buffers:  
  - hits: 3 (~24.00 KiB) from the buffer pool  
  - reads: 6 (~48.00 KiB) from the OS file cache, including disk I/O  
  - dirtied: 0  
  - writes: 0  
SELECT MAX("namespaces"."id") FROM "namespaces" INNER JOIN "audit_events_external_audit_event_destinations" ON "audit_events_external_audit_event_destinations"."namespace_id" = "namespaces"."id" WHERE "namespaces"."type" = 'Group'

 Aggregate  (cost=5.10..5.11 rows=1 width=4) (actual time=0.113..0.114 rows=1 loops=1)
   Buffers: shared hit=9
   I/O Timings: read=0.000 write=0.000
   ->  Nested Loop  (cost=0.56..5.10 rows=1 width=4) (actual time=0.109..0.111 rows=1 loops=1)
         Buffers: shared hit=9
         I/O Timings: read=0.000 write=0.000
         ->  Index Only Scan using index_external_audit_event_destinations_on_namespace_id on public.audit_events_external_audit_event_destinations  (cost=0.12..1.64 rows=1 width=8) (actual time=0.041..0.042 rows=1 loops=1)
               Heap Fetches: 0
               Buffers: shared hit=2
               I/O Timings: read=0.000 write=0.000
         ->  Index Scan using namespaces_pkey on public.namespaces  (cost=0.43..3.46 rows=1 width=4) (actual time=0.065..0.065 rows=1 loops=1)
               Index Cond: (namespaces.id = audit_events_external_audit_event_destinations.namespace_id)
               Filter: ((namespaces.type)::text = 'Group'::text)
               Rows Removed by Filter: 0
               Buffers: shared hit=7
               I/O Timings: read=0.000 write=0.000

Time: 0.529 ms  
  - planning: 0.372 ms  
  - execution: 0.157 ms  
    - I/O read: 0.000 ms  
    - I/O write: 0.000 ms  
  
Shared buffers:  
  - hits: 9 (~72.00 KiB) from the buffer pool  
  - reads: 0 from the OS file cache, including disk I/O  
  - dirtied: 0  
  - writes: 0  
SELECT COUNT(DISTINCT "namespaces"."id") FROM "namespaces" INNER JOIN "audit_events_external_audit_event_destinations" ON "audit_events_external_audit_event_destinations"."namespace_id" = "namespaces"."id" WHERE "namespaces"."type" = 'Group' AND "namespaces"."id" >= 22 AND "namespaces"."id" < 111

 Aggregate  (cost=5.39..5.40 rows=1 width=8) (actual time=0.096..0.097 rows=1 loops=1)
   Buffers: shared hit=19
   I/O Timings: read=0.000 write=0.000
   ->  Nested Loop  (cost=0.56..5.39 rows=1 width=4) (actual time=0.059..0.060 rows=0 loops=1)
         Buffers: shared hit=11
         I/O Timings: read=0.000 write=0.000
         ->  Index Only Scan using index_external_audit_event_destinations_on_namespace_id on public.audit_events_external_audit_event_destinations  (cost=0.12..1.64 rows=1 width=8) (actual time=0.020..0.021 rows=1 loops=1)
               Heap Fetches: 0
               Buffers: shared hit=2
               I/O Timings: read=0.000 write=0.000
         ->  Index Scan using namespaces_pkey on public.namespaces  (cost=0.43..3.46 rows=1 width=4) (actual time=0.035..0.035 rows=0 loops=1)
               Index Cond: ((namespaces.id = audit_events_external_audit_event_destinations.namespace_id) AND (namespaces.id >= 22) AND (namespaces.id < 111))
               Filter: ((namespaces.type)::text = 'Group'::text)
               Rows Removed by Filter: 0
               Buffers: shared hit=9
               I/O Timings: read=0.000 write=0.000

Time: 0.709 ms  
  - planning: 0.544 ms  
  - execution: 0.165 ms  
    - I/O read: 0.000 ms  
    - I/O write: 0.000 ms  
  
Shared buffers:  
  - hits: 19 (~152.00 KiB) from the buffer pool  
  - reads: 0 from the OS file cache, including disk I/O  
  - dirtied: 0  
  - writes: 0

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #344670 (closed)

Merge request reports