Add support for evaluating against the Duo Chat API
What does this merge request do and why?
Follow-up to https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/prompt-library/-/merge_requests/143.
- Make it config based. The cli then becomes:
poetry run promptlib duo-chat eval --config-file data/config/duochat_config.json.example
- Add support for the Duo Chat API.
- Make evaluation parsing more robust by searching and extracting a
<grades>
tag
How to set up and validate locally
Numbered steps to set up and validate the change are strongly suggested.
Merge request checklist
-
I've ran the affected pipeline(s) to validate that nothing is broken. -
Tests added for new functionality. If not, please raise an issue to follow up. -
Documentation added/updated, if needed.
Edited by Bruno Cardoso