VSM Dashboard - adding AI Descriptive Analytics ( “Duo insights”) with quick insights
Problem
In GitLab analytics there is simply too much data to analyze and software laders don't have enough time, resources, and talent to search for insights.
To increasing speed to insight software laders need explanations of data visualizations in plain language. For example:
- "After three months of improvement, Deployment frequency for project D dropped by 12% in February"
- "Cycle times for Group 1 and 2 show a similar pattern - after decreasing for two months, they began to increase in February."
- "All metrics went down in January, similar to the seasonal trend we saw in January 2022."
MVC Proposal
Adding AI Descriptive Analytics to the comparison metrics for detecting metrics outliers by answer this question:
"What metric had the most significant changes in the past month?"
Suggested MVC solution
- When user open the Duo chat on VSD page, adding a new option to "Get quick insights".
- When user asked for "Get quick insights" - search on the dataset for significant outliers.
- In this MVC, the outliers will be detected by calculating and identify the metric with the higher or lower Z-score.
Calculating the higher or lower Z-score:
- Import the month-over-month
percentage change
for each metric. - Calculate the
mean
andstandard deviation
of the month-over-month percentage change for each metric. Observation period is 6 months. - Calculate the Z-score for each metric:
(last_month_value - mean) / standard deviation = Z-score
. - Compare all the Z-scores and detect the metric with the higher or lower Z-score - this will be the answer for "What metrics had significant changes in the past month?"
- Add textual descriptions of the quick insights -
In <month>, the most significant changes was with <metric_name> for <project/group> <increasing/decreasing> by <percentage change> to <last_month_value>.
- E.g. "In February, the most significant changes was with Deployment frequency for project D increasing by +12% to 135.4/d"
Next MVC iterations:
Click to expand
Adding answers to also to these questions:
- Are there any metrics that show a similar pattern?
- What is the seasonality of my group performance?
Optional categories for AI-based Descriptive Analytics insights:
- Metrics outliers - detect when there are specific metrics with values significantly different than the other metrics.
- Correlation - Detects cases where multiple measures show a similar pattern or trend.
- Change points in a time- Highlights when there are significant changes in trends in a time series of data.
- Seasonality in time series - Finds periodic patterns in time series data, such as weekly, monthly, or yearly seasonality.
- Overall trends in time series - Detects upward or downward trends in time series data.
- Time outliers - For data across a time series, detects when there are specific dates or times with values significantly different than the other date/time values.
- Low Variance - Detects cases where data points for a dimension aren't far from the mean, so the variance is low.
- Majority (Major factors) - Finds cases where most of a total value can be attributed to a single factor when broken down by another dimension.
User experience goals:
Using Natural language generation creates textual descriptions of insights from the data and explanations of data visualizations. What happened and where, and when. For example: "In the last 3 months overall team velocity increased by 17%, while stability remain flat. There is a common Seasonality across these projects: A,B,C"