Skip to content

Support code explanation dataset on Duo Chat eval pipeline

Tan Le requested to merge add-code-explanation-config-template into main

What does this merge request do and why?

Support code explanation dataset on Duo Chat eval pipeline. This will address the current issue with the code explanation pipeline where we use the wrong prompt for Duo Chat.

Relates to #131 (closed)

How to set up and validate locally

  1. Ensure GCP environment variables are setup.
  2. Check out to this merge request's branch.
  3. Copy the example config.
    cp data/config/duochat_eval_code_explanation_config.json{.example,}
  4. Update the output_sinks to write to your prefer destinations.
    {
      "type": "bigquery",
      "path": "dev-ai-research-0e2f8974.duo_chat_experiments",
      "prefix": "tl_code_explanation_v3"
    }
  5. Run the follow command to kick off the pipeline.
    poetry run promptlib duo-chat eval --test-run --sample-size 1 --config-file=data/config/duochat_eval_code_explanation_config.json
  6. Inspect results from Duo Chat and LLM looks alright.

Merge request checklist

  • I've ran the affected pipeline(s) to validate that nothing is broken.
  • Tests added for new functionality. If not, please raise an issue to follow up.
  • Documentation added/updated, if needed.
Edited by Tan Le

Merge request reports