@ohoral Is this issue the reason why I do not see a model name for streamed suggestions in Tableau? I just see Null. I just created gitlab-org/gitlab#463792 (closed), but I think that is actually a duplicate of this issue.
The main issue is that the Client (LS) has no knowledge of the model being used. It can assume the language but not the model unless it is returned in the API response. But it is not returned for streamed responses. The response for the streamed and non-streamed respondes has a different format.
@partiaga First of all, a streamed response is always code generation. None of the telemetry events for streaming suggestion have "model" data just because it is not provided by the API. If this issue is done and the API returns the model-related data, we'll be able to track it.
Also, can I confirm with you that the context being set in different parts in the suggestion service has no relation to this problem?
this is exactly the place where the Snowplow telemetry event is enriched with model data if it was returned by the Code suggestions API along with the suggestion itself. Note, that for non-streamed suggestion events, suggestion_requested, suggestion_cancelled, and suggestion_error won't have model data because those events are tracked before API response is returned. Let me know if this helps or you have other questions or feel free to setup a sync if needed
this is exactly the place where the Snowplow telemetry event is enriched with model data if it was returned by the Code suggestions API along with the suggestion itself
I was looking at this part and thought so too, but the description in this issue is talking about something else:
...so I was confused. That line of code referred to in the description no longer exists, so I thought it was moved here. It seems it was moved to the suggestion service instead?
Note, that for non-streamed suggestion events, suggestion_requested, suggestion_cancelled, and suggestion_error won't have model data because those events are tracked before API response is returned.
Thanks, that's good to know!
Let me know if this helps or you have other questions or feel free to setup a sync if needed
I think you've cleared up my questions now.
I mostly wanted to see whether the Model Name = Null in the Acceptance Metrics is caused by anything else other than the streaming response, but it seems like that would be the only cause for it.
I discovered that this problem of Model Name = Null in Acceptance Metrics recently started happening for non-streamed responses (see gitlab-org/gitlab#498490 (comment 2200239300)). I am still investigating it now. I'll let you know once I have more info
@partiaga the error started happening in LS 6.8 or 6.10 v6.7.0...v6.8.0 - I do not see any changes that could have influenced that. In general, given it is so minor, I would not spend too much time and just keep an eye on it. One advice - when posting screenshots from Tableau - make the comment private as this is internal-only telemetry.
@dashaadu - would your team be able to prioritize this issue in 17.8? I am pushing to fully resolve the telemetry issues that have caused blind spots on streamed code generation events; e.g. inability to measure acceptance rate.
We have been working the core changes here, and I think the related LS changes are this issue along with #613 (closed) (which Angelo mentioned he could take on in 17.8). Happy to talk through details with you.