Skip to content

Add support for evaluating against the Duo Chat API

Bruno Cardoso requested to merge bc/duo-chat-api into main

What does this merge request do and why?

Follow-up to https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/prompt-library/-/merge_requests/143.

  • Make it config based. The cli then becomes:
poetry run promptlib duo-chat eval --config-file data/config/duochat_config.json.example
  • Add support for the Duo Chat API.
  • Make evaluation parsing more robust by searching and extracting a <grades> tag

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

Merge request checklist

  • I've ran the affected pipeline(s) to validate that nothing is broken.
  • Tests added for new functionality. If not, please raise an issue to follow up.
  • Documentation added/updated, if needed.
Edited by Bruno Cardoso

Merge request reports

Loading