Skip to content

Add intent param to cli to run code generation

Missy Davies requested to merge md-code-gen-intent into main

What does this merge request do and why?

Add an arg to the cli in order to force using either intent = generation or completion. This allows us to run code generation explicitly against a dataset made for code generation.

Resolves #7 (closed)

A note on the dataset

code_generation_mbpp_all_sanitized

I downloaded this from BigQuery then renamed the columns to match our inputs/output. It should be the same as the code_gen_chat_dataset dataset, except the latter does not have the right inputs/output to be able to run our existing code suggestions evaluate.py.

How to set up and validate locally

poetry run eli5 code-suggestions evaluate \
  --dataset="code_generation_mbpp_all_sanitized" \
  --source=gitlab \
  --limit=10 \
  --offset=5 \
  --evaluate-with-llm \
  --experiment-prefix=exp \
  --rate-limit=100 \
  --intent="generation"

Merge request checklist

  • Tests added for new functionality. If not, please raise an issue to follow up.
  • Documentation added/updated, if needed.
Edited by Missy Davies

Merge request reports