Update return type of ModelEnginePalm: follow-up from "Improve code-gecko token length estimations"
The following discussion from !217 (merged) should be addressed:
-
@stanhu started a discussion: (+1 comment) This can be separate from this merge request, but I noticed we're missing a return type here.
I was thinking we might want to return a more generic object so we can capture:
- The completion itself
- Estimated token count
- Number of tokens truncated in prefix
- Number of tokens truncated in suffix
- Which prompt transformations were applied (e.g. imports)
Then we can log this.