Add intent param to cli to run code generation
What does this merge request do and why?
Add an arg to the cli in order to force using either intent = generation or completion. This allows us to run code generation explicitly against a dataset made for code generation.
Resolves #7 (closed)
A note on the dataset
code_generation_mbpp_all_sanitized
I downloaded this from BigQuery then renamed the columns to match our inputs/output. It should be the same as the code_gen_chat_dataset dataset, except the latter does not have the right inputs/output to be able to run our existing code suggestions evaluate.py.
How to set up and validate locally
poetry run eli5 code-suggestions evaluate \
--dataset="code_generation_mbpp_all_sanitized" \
--source=gitlab \
--limit=10 \
--offset=5 \
--evaluate-with-llm \
--experiment-prefix=exp \
--rate-limit=100 \
--intent="generation"
Merge request checklist
-
Tests added for new functionality. If not, please raise an issue to follow up. -
Documentation added/updated, if needed.
Edited by Missy Davies