AI Usage Dashboard
Update:
This was our workflowproblem validation issue.
Here is the workflowsolution validation epic with mapping of the problems and possible solutions for adding "AI Impact" to Category:Value Stream Management. More feedback is welcome in that epic.
Customer request from @mfinch:
Do you know if there is anyway to see how many people are utilizing the coding suggestions api beta in an org or even by repo which have it used for?
Idea:
- new Value Stream management report for users of AI features and impacts to productivity metrics
- Start with just how many are using what AI features.
- Metric ideas
- Number of AI actions
- Estimations of time saved (faster deployments, faster time to resolve MRs, others?0
- % of users using AI features
- Comparison of users using AI vs none
Existing VSM dashboard: https://gitlab.com/groups/gitlab-org/-/analytics/dashboards/value_streams_dashboard?query=gitlab-org/gitlab
Take these existing metrics and filter them based on user's using AI features
We already have service ping datapoints
these events AND ((event_category = 'Llm::ExecuteMethodService' --tracking for all but Code Suggestions AND event_action = 'execute_llm_method') OR event_category IN ('API::Ml::AiAssist', 'API::CodeSuggestions')) --Code Suggestions has slightly different tracking
*/
SELECT
*,
f.value['schema']::VARCHAR AS context_data_schema,
f.value['data'] AS context_data
FROM common_mart.mart_behavior_structured_event,
lateral flatten(input => TRY_PARSE_JSON(contexts), path => 'data') f
WHERE app_id = 'gitlab_ai_gateway' AND behavior_at >= '2023-09-01'
) , suggestions_with_user_id AS ( SELECT \*, context_data\['gitlab_global_user_id'\] as gitlab_global_user_id, context_data\['gitlab_realm'\]::VARCHAR as gitlab_realm FROM suggestions_with_context WHERE context_data_schema like 'iglu:com.gitlab/code_suggestions_context/jsonschema/%' )https://docs.gitlab.com/ee/user/ai_features.html
Really rough vision of how this could manifest itself in VSM