## Problem Currently we have under the [Naming and placing the metrics](https://docs.gitlab.com/ee/development/usage_ping.html#1-naming-and-placing-the-metrics) ``` Add the metric in one of the top level keys license: for license related metrics. settings: for settings related metrics. counts_weekly: for counters that have data for the most recent 7 days. counts_monthly: for counters that have data for the most recent 28 days. counts: for counters that have data for all time. ``` Opening this issue to discuss about the convention for the key path of a metric in JSON payload There are a couple of exceptions when we currently add new metrics. For example Redis counters go under same redis_hll_couters top level key The current usage ping structure and naming convention is vague: - there is a loosely defined structure of how to organize/nest counters, making it difficult to know where to put new counters. - the naming convention is not standardized across all counters, making it difficult to know what to name new counters. ## Nesting structure The goal is to keep this as flat as possible and instead to add details into the naming convention or event dictionary. - license - settings - counts_7d - counts_28d - counts_all ## Naming convention of metrics and events: - top_level: count_7d, count_28d, count_all, license, setting - segmentation: group, project, user, none - event_name: primary name in [event dictionary](https://docs.google.com/spreadsheets/d/1VzE8R72Px_Y_LlE3Z05LxUlG_dumWe3vl-HeUo70TPw/edit#gid=618391485) For example: - `#{top_level}.#{segmentation}_#{event_name}` - count_all.projects_prometheus_active ## Event Dictionary: We continue to add to the existing event dictionary: https://docs.google.com/spreadsheets/d/1VzE8R72Px_Y_LlE3Z05LxUlG_dumWe3vl-HeUo70TPw/edit#gid=618391485 - product_owner: [section](https://gitlab.com/gitlab-com/www-gitlab-com/-/blob/master/data/sections.yml), [stage](https://gitlab.com/gitlab-com/www-gitlab-com/blob/master/data/stages.yml), [group](https://gitlab.com/gitlab-com/www-gitlab-com/blob/master/data/stages.yml) - product_category: [categories.yml](https://gitlab.com/gitlab-com/www-gitlab-com/blob/master/data/categories.yml) - implementation: snowplow js, snowplow ruby, redis hll, ordinary batch counter, etc. - metric and event name - description - etc. For example: | metric and event name | product owner | product category | implementation | ... | | ------ | ------ | ------ | ------ | ------ | | count_all.projects_prometheus_active | ops.monitor | integrations | usage ping batch counting | ... | ## Migration Plan - Create a v2 Usage Ping file and job that runs in parallel to the existing Usage Ping. Try to combine both CE and EE files together. - Create a v2 Versions endpoint to collect Usage Ping. Write the raw usage data to a separate table. - Run both v1 and v2 in parallel for two milestones. - In the data warehouse, use dbt to transform the v1 structure to v2. This will ensure all previous data ends up in v2. - Update all Sisense charts to read from v2 tables. ## Next steps - [ ] Discuss proposal and ensure all use cases are covered - [ ] Create data migration plan - [ ] Update to new convention - [ ] Proposal for top level keys https://gitlab.com/gitlab-org/gitlab/-/issues/251011 - [ ] Add `metrics.yml` file under `lib/gitlab/usage_data` https://gitlab.com/gitlab-org/gitlab/-/issues/270107 cc @gitlab-org/growth/product-intelligence/engineers