Skip to content

Draft: feat: add cli param to run dataset splits

Missy Davies requested to merge md-cli-param-splits into main

What does this merge request do and why?

Resolves #22

TODO:

  • Add class for only existing splits
  • See what happens running multiple
  • Change name to singular?

How to set up and validate locally

Existing splits are listed at the top of the code-suggestions-input-testcases-v1 dataset:

Screenshot_2024-07-11_at_12.30.29_PM

poetry run eli5 code-suggestions evaluate \
  --dataset="code-suggestions-input-testcases-v1" \
  --source=gitlab \
  --limit=10 \
  --offset=0 \
  --experiment-prefix=splits-test \
  --intent=completion \
  --rate-limit=29 \
  --splits=lang=php

Merge request checklist

  • Tests added for new functionality. If not, please raise an issue to follow up.
  • Documentation added/updated, if needed.
Edited by Missy Davies

Merge request reports