Skip to content

Raise exception when encountering errors from gitlab client

Pam Artiaga requested to merge pam/raise-an-exception-for-errors into main

What does this merge request do and why?

Raise exception when encountering errors from gitlab client.

These raised exceptions will not stop the script, but it marks the example as having an error in the Langsmith evaluation result

How to Validate / Screenshots

  1. Make your GitLab /codesuggestions endpoint return an error.

    For example, make sure it has a rate limit, and on your evaluation script, run an evaluation over 100 examples that is not rate limited. This will ensure that the GitLab endpoint will return a 429 - too many requests error.

  2. Run the evaluation script as described above:

    poetry run eli5 code-suggestions evaluate \
      --dataset="code-suggestions-input-testcases-v1" \
      --source=gitlab \
      --limit=500 \
      --offset=0 \
      --evaluate-with-llm \
      --experiment-prefix=pam-test-100
  3. Verify that the script still runs until the end

  4. Verify that the errors from the GitLab client are recorded in the Langsmith evaluation result:

    Error rate

    Screenshot_2024-07-03_at_16.37.48

    Error details in the example

    Screenshot_2024-07-03_at_16.36.52

Merge request checklist

  • Tests added for new functionality. If not, please raise an issue to follow up.
  • Documentation added/updated, if needed.

Related issue: Evaluation script improvements (#16 - closed)

Edited by Pam Artiaga

Merge request reports