Skip to content

Move code-suggestions eval logic to the eli5 package

Alexander Chueshev requested to merge ac/move-codesuggestions-eli5 into main

This MR moves the existing code-suggestions evaluation logic the eli5 package. The idea is to consolidate our development following the Python standards. Please, note this MR doesn't change the logic and provides the same CLI API using the typer package.

How to run:

  1. Set the required env variables by creating a .env file in the root project folder or manually:
    ELI5_LANGCHAIN_API_KEY="your_langsmith_api_key" # renamed from LANGCHAIN_API_KEY
  2. Get help by running:
    poetry install # install all required dependencies
    poetry run eli5 code-suggestions --help
    poetry run eli5 code-suggestions evaluate --help
  3. Run the evaluation process:
    poetry run eli5 code-suggestions evaluate \
      --dataset="<your dataset>" \
      --source=gitlab \
      --num-tests=0 \
      --evaluate-with-llm \
      --experiment-prefix=experiment
Edited by Alexander Chueshev

Merge request reports