Skip to content

Introduce filtering for total count metrics

Jonas Larsen requested to merge 435338-introduce-filter-by-additional-properties-split-1 into master

What does this MR do and why?

Introduce filtering on additional properties for total count metrics

Total count metrics can now be defined with a filter which decides if an event will update a metric or not.

Here is an example of a metric definition using filters. The count of the metric will be increased if a perform_epics_action event with a label value of abc is emitted or if a perform_issues_action with a label value of xyz and a property value of hello:

key_path: counts.count_total_perform_epics_action_label_abc_or_label_xyz_monthly
description: 'Mountly count of some events. This description should be better'
product_section: dev
product_stage: plan
product_group: product_planning
performance_indicator_type: []
value_type: number
status: active
milestone: '17.0'
introduced_by_url: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/151646
time_frame: 28d
data_source: internal_events
data_category: optional
distribution:
- ee
tier:
- premium
- ultimate
events:
- name: perform_epics_action
  filter:
    label: abc
- name: perform_issues_action
  filter:
    label: xyz
    property: hello

Since only additional properties can be used in filters, the only valid keys for a filter are label, property and value.

The main focus has been to support the migration of Redis metrics to internal events like decribed here: #424893

Future work

This future work will continue immediately after this MR is merged.

  • Documentation will be added in a separate MR when we migrate the first metrics
  • Add filters for RedisHLL metrics
  • Add filter support to the generator

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

How to set up and validate locally

  1. Create a few different metric definitions using the same event and different filters. You can get some like this curl -s https://gitlab.com/gitlab-org/gitlab/-/commit/a6324efd6771bdac44654789178db064e1d39c03.diff | git apply -
  2. Start the monitor: rails runner scripts/internal_events/monitor.rb perform_epics_action
  3. Start gdk and run the below commands one at a time and see the values in the monitor update as you expect.
Gitlab::InternalEvents.track_event('perform_epics_action', additional_properties: {label: 'abc'})
Gitlab::InternalEvents.track_event('perform_epics_action', additional_properties: {label: 'xyz'})
Gitlab::InternalEvents.track_event('perform_epics_action')
Gitlab::InternalEvents.track_event('perform_epics_action', additional_properties: {label: 'foo'})

Afterwards it should look like this:

Updated at 2024-05-02 07:55:36 UTC [PAUSED]
Monitored events: perform_epics_action

+-------------------------------------------------------------------------------------------------------------------------------------------------------+
|                                                                   RELEVANT METRICS                                                                    |
+------------------------------------------------------------------------+----------------------+-----------------------+---------------+---------------+
| Key Path                                                               | Monitored Events     | Instrumentation Class | Initial Value | Current Value |
+------------------------------------------------------------------------+----------------------+-----------------------+---------------+---------------+
| counts.count_total_perform_epics_action_label_abc_monthly              | perform_epics_action | TotalCountMetric      | 0             | 1             |
| counts.count_total_perform_epics_action_label_abc_or_label_xyz_monthly | perform_epics_action | TotalCountMetric      | 0             | 2             |
| counts.count_total_perform_epics_action_label_xyz_monthly              | perform_epics_action | TotalCountMetric      | 0             | 1             |
| counts.count_total_perform_epics_action_monthly                        | perform_epics_action | TotalCountMetric      | 0             | 4             |
+------------------------------------------------------------------------+----------------------+-----------------------+---------------+---------------+
+----------------------------------------------------------------------------------------------------------------------+
|                                                   SNOWPLOW EVENTS                                                    |
+----------------------+--------------------------+-----------------------+---------+--------------+------------+------+
| Event Name           | Collector Timestamp      | Category              | user_id | namespace_id | project_id | plan |
+----------------------+--------------------------+-----------------------+---------+--------------+------------+------+
| perform_epics_action | 2024-05-02T07:55:32.657Z | InternalEventTracking |         |              |            |      |
| perform_epics_action | 2024-05-02T07:54:40.166Z | InternalEventTracking |         |              |            |      |
| perform_epics_action | 2024-05-02T07:53:41.407Z | InternalEventTracking |         |              |            |      |
| perform_epics_action | 2024-05-02T07:52:13.917Z | InternalEventTracking |         |              |            |      |
+----------------------+--------------------------+-----------------------+---------+--------------+------------+------+

Numbered steps to set up and validate the change are strongly suggested.

Related to #435338

Edited by Jonas Larsen

Merge request reports