Fetch the 'text' answer from the GitLab AI response
Currently, when fetching code suggestions from the GitLab AI model (evaluation_scripts/code_suggestions/clients/gitlab.py
), the response we get is:
{'output': {'text': 'the code suggestion here', 'index': 0, 'finish_reason': 'length'}}
In the exact_match
evaluator, what we're trying to check is:
"prediction": run.outputs.get("output", "")
If I'm not mistaken, we want this prediction
to be the actual text
prediction without the other fields (index
and finish_reason
).
This MR should align the response of gitlab_get_code_suggestion
with what's being set as the prediction in the exact match evaluator. We also change the llm-as-judge evaluator to match the precedent set by the exact_match
.
Edited by Pam Artiaga