Skip to content

Fetch the 'text' answer from the GitLab AI response

Pam Artiaga requested to merge pam/align-fetched-outputs-in-code-suggestions into main

Currently, when fetching code suggestions from the GitLab AI model (evaluation_scripts/code_suggestions/clients/gitlab.py), the response we get is:

{'output': {'text': 'the code suggestion here', 'index': 0, 'finish_reason': 'length'}}

In the exact_match evaluator, what we're trying to check is:

"prediction": run.outputs.get("output", "")

If I'm not mistaken, we want this prediction to be the actual text prediction without the other fields (index and finish_reason).

This MR should align the response of gitlab_get_code_suggestion with what's being set as the prediction in the exact match evaluator. We also change the llm-as-judge evaluator to match the precedent set by the exact_match.

Edited by Pam Artiaga

Merge request reports