Raise exception when encountering errors from gitlab client
What does this merge request do and why?
Raise exception when encountering errors from gitlab
client.
These raised exceptions will not stop the script, but it marks the example as having an error in the Langsmith evaluation result
How to Validate / Screenshots
-
Make your GitLab
/codesuggestions
endpoint return an error.For example, make sure it has a rate limit, and on your evaluation script, run an evaluation over
100
examples that is not rate limited. This will ensure that the GitLab endpoint will return a429 - too many requests error
. -
Run the evaluation script as described above:
poetry run eli5 code-suggestions evaluate \ --dataset="code-suggestions-input-testcases-v1" \ --source=gitlab \ --limit=500 \ --offset=0 \ --evaluate-with-llm \ --experiment-prefix=pam-test-100
-
Verify that the script still runs until the end
-
Verify that the errors from the GitLab client are recorded in the Langsmith evaluation result:
Error rate
Error details in the example
Merge request checklist
-
Tests added for new functionality. If not, please raise an issue to follow up. -
Documentation added/updated, if needed.
Related issue: Evaluation script improvements (#16 - closed)