馃摵 Usage Ping with AMAU and updated SMAU - Parent Issue
Background and Overview
The current SMAU dashboards are incorrect:
-
Self managed - The current reported SMAU is overestimated as it does not count unique users performing actions within a stage. Currently, If 1 user uses 1 feature from a stage, we count every user on the instance as a Monthly Active User for that stage. Usage ping is also struggling to do basic SELECT COUNT(*) on large instances; many of these queries time out on GitLab.com
-
.Com - SMAU is currently blocked due to:
- User_id is not available for snowplow events
- Structured events are not in place. Currently we have unstructured, self-describing events, which requires parsing through the unstructured data and the process will break whenever the CSS or JS changes on the site, there is no stability in these types of events
The proposal is to fix the time out issue for larger instances on usage ping and update the SMAU definition to be highest AMAU (Actions Monthly Active User) for a stage. The .com SMAU and AMAU will also be based on the usage ping data.
Outcomes
- Improve accuracy of SMAU
- Improve completeness of usage ping dataset
- Internally consistent view of SMAU between self-managed and .com
- Measure active use in a Periscope dashboard
- Enable us to improve SMAU
Other Resources
Problems To Solve:
- A) Fix usage ping: Usage ping has been broken on GitLab.com since Jan 2019 due to an uncaught error.
- B) Stage specific counters: Some stages currently do not have usage ping counters.
- C) Query optimizations: Usage ping counters are timing out due to very large tables.
- D) Dashboards: All of this data is being used to create SMAU dashboards in Sisense.
- E) Documentation: The current documentation is lacking making it hard for stage teams to implement usage ping.
- F) Harden Usage Ping: Usage ping in it's current state is fragile and we need to increase it's stability.
Next Steps
--- 12.9 ---
A) Fix usage ping:
-
0. Usage ping timing out for larger instances https://gitlab.com/gitlab-org/telemetry/issues/264 -
1. Time period for usage activity by stage in usage data gitlab-org/gitlab#201764 (closed) -
2. Ensure gitlab.com usage ping payload reaches snowflake https://gitlab.com/gitlab-org/telemetry/issues/299 -
2.1. Fix avg_cycle_analytics giving an uncaught error gitlab-org/gitlab#208788 (closed) gitlab-org/gitlab!26381 (merged) -
2.2. Disable usage ping on .com to avoid collisions https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/9385 -
2.3. Make sure usage ping is fully processed on version app https://gitlab.com/gitlab-org/telemetry/-/issues/348 -
2.3.1 Add Monthly Stage Usage Data to Versions https://gitlab.com/gitlab-services/version-gitlab-com/-/issues/280 https://gitlab.com/gitlab-services/version-gitlab-com/-/merge_requests/390
-
-
2.4. Increase manual import frequency https://gitlab.com/gitlab-data/analytics/-/issues/4040
-
-
3. Test Usage Ping on Self-Hosted Instances https://gitlab.com/gitlab-org/telemetry/issues/309 -
4. Check each query conditions gitlab-org/gitlab#205281 (closed) -
5. Remove Feature Flags and Release to Self-Hosted Instances https://gitlab.com/gitlab-org/telemetry/issues/306
B) Stage specific counters:
-
Secure: Update Secure usage ping to be able to determine number of jobs by user gitlab-org/secure/general#63 (closed) gitlab-org/gitlab#208697 (closed) -
Fix Secure queries returning -1's (timeouts).
-
C) Query optimizations:
-
Fix priority counters on GitLab.com, all these should work on self-managed with or without the optimization fixes
Milestone | Status | Assignee | Weight | Title | Issue | Merge request |
---|---|---|---|---|---|---|
12.9 | verified working on gprd | @a_akgun | 3 | Optimize todos counters | gitlab-org/gitlab#208891 (closed) | gitlab-org/gitlab!26442 (merged) |
12.9 | verified working on gprd | @a_akgun | 3 | Optimize notes and suggestions counters | gitlab-org/gitlab#208890 (closed) | gitlab-org/gitlab!26871 (merged) |
12.9 | verified working on gprd | @a_akgun | 3 | Optimize event counters | gitlab-org/gitlab#208889 (closed) | gitlab-org/gitlab!26444 (merged) |
12.9 | verified working on gprd | @a_akgun | 3 | Optimize service_desk_issue counters related to the ::Issue | gitlab-org/gitlab#208888 (closed) | gitlab-org/gitlab!26445 (merged) |
12.9 | verified working on gprd | @alinamihaila | 3 | Optimize deployment counters related to the ::Deployment | gitlab-org/gitlab#208886 (closed) | gitlab-org/gitlab!26757 (merged) |
12.9 | verified working on gprd | @dstull | 3 | Optimize project counters related to the ::Project | gitlab-org/gitlab#208887 (closed) | gitlab-org/gitlab!26686 (merged) gitlab-org/gitlab!26698 (merged) gitlab-org/gitlab!26802 (merged) gitlab-org/gitlab!27115 (merged) |
12.9 | verified working on gprd | @a_akgun | 3 | Optimize ci_pipeline counters related to the ::Ci::Pipeline | gitlab-org/gitlab#208885 (closed) | gitlab-org/gitlab!26774 (merged) |
12.9 | verified working on gprd | @a_akgun | 3 | Optimize services_usage counters | gitlab-org/gitlab#210007 (closed) | gitlab-org/gitlab!26973 (merged) |
12.9 | verified working on gprd | @alinamihaila | 5 | Approximate counters are not working on gitlab.com | gitlab-org/gitlab#210332 (closed) | gitlab-org/gitlab!27218 (merged) |
12.9 | removed, duplicate | @alinamihaila | 3 | Optimize protected branch counters | gitlab-org/gitlab#210052 (closed) | gitlab-org/gitlab!27017 (closed) |
E) Documentation:
-
Clarify attributes in Usage Ping https://gitlab.com/gitlab-org/telemetry/issues/267
--- 12.10 ---
A) Fix usage ping:
-
Reenable automated usage ping on .com
C) Query optimizations:
-
Fix priority counters on GitLab.com, all these should work on self-managed with or without the optimization fixes
D) Dashboards:
-
GitLab.com Postgres SMAU https://app.periscopedata.com/app/gitlab/604621/GitLab.com-Postgres-SMAU gitlab-org/growth/product#1484 -
GitLab.com + Self Managed Usage Ping SMAU https://app.periscopedata.com/app/gitlab/634200/WIP-GitLab.com-+-Self-Managed-Usage-Ping-SMAU https://gitlab.com/gitlab-data/analytics/-/issues/4124 -
Usage Ping QA: https://app.periscopedata.com/app/gitlab/632033/WIP:-Usage-Ping-QA https://gitlab.com/gitlab-data/analytics/-/issues/4144 -
Dashboard to verify usage ping counter fallbacks per version number https://gitlab.com/gitlab-data/analytics/-/issues/4064 -
Analyze usage ping on 12.10 https://gitlab.com/gitlab-org/gitlab/issues/208233
-
F) Harden usage ping:
-
Harden Usage Ping https://gitlab.com/gitlab-org/telemetry/-/issues/335 - Isolate usage ping for robustness
-
12.10 Consolidate all counters into four main counters with fail safes gitlab-org/gitlab#212403 (closed) -
Isolate Redis Counters gitlab-org/gitlab#212402 (closed) -
Isolate Alternative Counter gitlab-org/gitlab#212400 (closed) -
Convert Counter: Any left over counters convert into batch counters gitlab-org/gitlab#208923 (closed) -
Convert Counter: Jira Usage gitlab-org/gitlab#212398 (closed) -
Convert Counter: cycle_analytics
gitlab-org/gitlab#212401 (closed)
-
-
12.10 Create an example of add_usage_data
method.-
add_usage_data
method w/ four examples
-
-
- Make Versions more robust
-
12.10 Save original usage ping payload before parsing https://gitlab.com/gitlab-services/version-gitlab-com/-/issues/281
-
- Isolate usage ping for robustness
--- 13.0 ---
F) Harden usage ping:
-
Harden Usage Ping https://gitlab.com/gitlab-org/telemetry/-/issues/335 - Isolate usage ping for robustness
-
13.0 Expand add_usage_data
to all 400 counters
-
- Isolate usage ping for robustness