AI impact dashboard- customer facing using VSA
Overview
As we expend our AI capabilities and release more future to GA, our customer would want to know the usage and the impact that AI is creating in their organizations, since we are not making our tableau dashboards public, and not every metric we have is relevant to our customers a natural candidate can be utilizing the VSA for that.
Metrics and details
Principles
- The dashboard should reflect the potential of usage. For example, 10 users who engaged with a feature is not deterministically good or bed. If its 10/11 its great if it's 10/1000 it's probably not that great
- The dashboard should reflect objective numbers as well a % of engagement
- The dashboard should provide a view that shows usage over time
Candidate metrics
| Metric name | What to measure | Visualization |
|---|---|---|
| Potential active users | How many people have to AI add-on | A number in a widget |
| Active user over time | Number of unique active users over a chosen period of time |
Line chart where X stands for time and Y stands for number of people |
| Value unit metric |
The occurrent of the actual job the feature is doing for a chosen period of time i.e- code generated, Questions asked for chat etc. Here it makes sense that each team will decide on what represent the single value unit of their feature |
Line or bar chart where X stands for time and Y stands for the number of value unites happening |
| TBD |
Implementation details
Not all customers will use all of the features, and not all customers will provide access at every level, the analytics should reflect and consider that
We could have 2 approaches here
- Dashboard only reflect features that are activated or been used
- Dashboard reflects all features with empty states for the features that are not active
In any case the dashboard should be available only at the level that AI itself is active and not banned
Risks
Not all metrics that customers would like to see are actually a measure of success and some can be confusing or paint the wrong picture and we should consider that as we release
Competitive landscape
GitHub co-pilot
community discussion -https://github.com/orgs/community/discussions/54691