Add Gemini Pro 1.5 model support
What does this merge request do and why?
This adds support to evaluate with Gemini Pro 1.5.
Relates to #183 (closed)
How to set up and validate locally
-
Ensure GCP environment variables are setup.
-
Check out to this merge request's branch.
-
Use the following config file, named it
duochat_eval_config_gemini.json
.{ "beam_config": { "pipeline_options": { "runner": "DirectRunner", "project": "dev-ai-research-0e2f8974", "region": "us-central1", "temp_location": "gs://prompt-library/tmp/", "save_main_session": false } }, "input_bq_table": "dev-ai-research-0e2f8974.duo_chat_external.experiment_code_generation__input_v1", "output_sinks": [ { "type": "local", "path": "data/output", "prefix": "experiment" } ], "throttle_sec": 0.1, "batch_size": 10, "input_adapter": "mbpp", "eval_setup": { "answering_models": [ { "name": "gemini-1.5-pro-preview-0215", "prompt_template_config": { "templates": [ { "name": "empty", "template_path": "data/prompts/duo_chat/answering/empty.txt.example" } ] } } ], "metrics": [ { "metric": "independent_llm_judge", "evaluating_models": [ { "name": "text-bison@latest", "prompt_template_config": { "templates": [ { "name": "claude-2", "template_path": "data/prompts/duo_chat/evaluating/claude-2.txt.example" } ] } } ] } ] } }
-
Run the follow command to kick off the pipeline.
poetry run promptlib duo-chat eval --test-run --sample-size 1 --config-file=data/config/duochat_eval_config_gemini.json
-
View the perfect response
🎉 experiment_20240307_143350__independent_llm_judge-00000-of-00001.csv
Merge request checklist
-
I've ran the affected pipeline(s) to validate that nothing is broken. -
Tests added for new functionality. If not, please raise an issue to follow up. -
Documentation added/updated, if needed.
Edited by Tan Le