Skip to content

[VSD] - Adding a new metric - "AI (Code Suggestions) Usage Rate"

Problem

Engineering leadership (Director/VP/CTO) need to demonstrate the ROI from the investments in AI features. To quantify the impact of adding AI and to demonstrate the ROI, they need:

Proposal

  1. In Log First Daily Use of Code Suggestions by User (#441485 - closed) groupcode creation will record the first time during a day, a user receives a suggestion.
  2. groupoptimize will aggregate this record into Monthly Unique Code Suggestions users.
  3. The Monthly Unique Code Suggestions users will be stored in PS, and can be later cloned into CH.
  4. Using the new Monthly Unique Code Suggestions users calculate a new metric - "Monthly Code Suggestions Usage rate" Monthly Unique Code Suggestions users / Total Monthly unique contributors).
  5. Results will be generated for each group, sub-group and project.
  6. Next iteration will be UX: [VSD] "AI Impact view" MVC - visualize the ... (#443698 - closed) (frontend)

Code Suggestions unique users Usage rate vs Acceptance Rate %

For the MVC, we believe that comparing the Code Suggestions usage rate month-over-month will give a more accurate indication of AI Impact than Acceptance Rate %, since Acceptance Rate % will be easily impacted by:

  • The experience level of the developer, e.g. Senior eng will have a low acceptance rate, while joiners will have high.
  • The type of project and the level of complexity will also impact the quality of the suggestion.
  • Network latency will also be a factor.
Old proposal

Can we use the Code Suggestions Usage data to enrich the groupoptimize ClickHouse-based contributors table with a new "AI contributor event"?

Meaning that we will track on CH, the number of users that use the code suggestions. Once we have this enrichments, we can calculated the monthly "AI Usage rate" - Monthly Unique Code Suggestions users / Total Monthly unique contributors).

And can we compare this new "AI Usage rate" with other SDLC performance metrcis.

Edited by Haim Snir