Skip to content

Add unique property to Internal Events Tracking

Jonas Larsen requested to merge 409993-create-metric-definition-service into master

What does this MR do and why?

In Gitlab Internal Events Tracking we want to be able to express which property value is used for distinct counts. It might be user_id, project_id etc.

Until now we didn't have a way to express which property is used in the metric definition itself. This is making it impossible to use the metric definitions to reproduce the metrics based on Snowplow events in the data warehouse.

By adding the unique property to the metric definition, down stream systems can now reproduce the metrics.

In order not to break the down stream systems, everything under options is unchanged and a new root property (events) has been added. That means that the event information is duplicated while me migrate the next couple of metrics. We want to remove this duplication when the downstreams systems are ready for it and before migrating the rest of the metrics.

These changes are adding about 0.5 second (on my machine) to the startup time because it parses all the metric definitions. This is not optimal, but I hope that it is acceptable as a first iteration. Reducing the startup time has been extracted to a separate issue: #414653 (closed)

Limitations:

  1. Currently we only allow an event to have one unique key globally. This is due to the fact that we only create one RedisHLL key per event. We need a RedisHLL key per unique key to support using several unique properties.

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

Before After

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

  • Check with Data Engineering that the new format won't cause any problems in the data warehouse.
  • I have evaluated the MR acceptance checklist for this MR.

Related to #409993 (closed)

Edited by Jonas Larsen

Merge request reports