Skip to content

Move the Duo Chat evaluation scripts to the `eli5` package

Alexander Chueshev requested to merge ac/move-duo-chat-eli5 into main

What does this merge request do and why?

This MR moves the Duo Chat evaluation scripts to the eli5 package. As we move forward and add more complex logic, we need to follow Python standards, avoid code duplication, and have a common user CLI.

Please, note that:

  • we already moved the code suggestions evaluation scripts to the eli5 package - cc @acook.gitlab
  • we're able to run the scripts via the CLI commands
  • this MR doesn't improve the code quality. The goal is to bring everything under the same standards.

Moved scripts:

  • evaluate.py -> poetry run eli5 duo-chat evaluate-qa
  • evaluate_multi_dimension.py -> poetry run eli5 duo-chat evaluate-qa --evaluate-with-llm
  • evaluate_multi_dimension_with_custom_prompt.py -> poetry run eli5 duo-chat evaluate-qa --with-llm-custom_prompt
  • evaluate_performance.py -> poetry run eli5 duo-chat evaluate-performance
  • evaluate_documentation_questions_with_custom_prompt.py -> poetry run eli5 duo-chat evaluate-docs

How to set up and validate locally

  1. Check out to this merge request's branch and set the required env variables:
    cp .env.example .env
  2. Install dependencies.
    poetry run install
  3. Check the existing commands ELI5 provides:
    poetry run eli5 duo-chat --help

Merge request checklist

  • Tests added for new functionality. If not, please raise an issue to follow up.
  • Documentation added/updated, if needed - here is ongoing the MR already - !60 (merged)
Edited by Alexander Chueshev

Merge request reports