Usage Ping Restructure
Problem
The current usage ping structure and naming convention is vague:
- there is a loosely defined structure of how to organize/nest counters, making it difficult to know where to put new counters.
- the naming convention is not standardized across all counters, making it difficult to know what to name new counters.
Nesting structure
The goal is to keep this as flat as possible and instead to add details into the naming convention or event dictionary.
- license
- settings
- count_7d
- count_28d
- count_all
Naming convention of metrics and events:
- top_level: count_7d, count_28d, count_all, license, setting
- segmentation: group, project, user, none
- event_name: primary name in event dictionary
For example:
#{top_level}.#{segmentation}_#{event_name}
- count_all.projects_prometheus_active
Event Dictionary:
We continue to add to the existing event dictionary: https://docs.google.com/spreadsheets/d/1VzE8R72Px_Y_LlE3Z05LxUlG_dumWe3vl-HeUo70TPw/edit#gid=618391485
- product_owner: section, stage, group
- product_category: categories.yml
- implementation: snowplow js, snowplow ruby, redis hll, ordinary batch counter, etc.
- metric and event name
- description
- etc.
For example:
metric and event name | product owner | product category | implementation | ... |
---|---|---|---|---|
count_all.projects_prometheus_active | ops.monitor | integrations | usage ping batch counting | ... |
Migration Plan
- Create a v2 Usage Ping file and job that runs in parallel to the existing Usage Ping. Try to combine both CE and EE files together.
- Create a v2 Versions endpoint to collect Usage Ping. Write the raw usage data to a separate table.
- Run both v1 and v2 in parallel for two milestones.
- In the data warehouse, use dbt to transform the v1 structure to v2. This will ensure all previous data ends up in v2.
- Update all Sisense charts to read from v2 tables.
Next steps
-
Discuss proposal and ensure all use cases are covered -
Create data migration plan -
Update to new convention -
Proposal for top level keys gitlab-org/gitlab#251011 (closed) -
Add metrics.yml
file underlib/gitlab/usage_data
gitlab-org/gitlab#270107 (closed)
Edited by Alina Mihaila