Code Suggestions Telemetry - Include Model Name with Streamed Responses
Currently, when I look in Tableau, I see behavior that seems to indicate that along with streamed responses, we are not sending the model name (and possibly other information) like we do with non-streamed responses.
For example, on the Quality Metrics chart, I get the error that "Acceptance Rate no available by Model Name for streamed suggestions".
In other places it looks like the model name is Null
for streamed suggestions:
We should send the model name for better reporting so we can measure acceptance rate and latency (stream start time) by model name. This is important now because we are testing multiple code generation models and would like to be able to distinguish them on the dashboards.
Edited by Matt Nohr