Skip to content

Add Gemini Pro 1.5 model support

Tan Le requested to merge add-gemini-pro-1-5 into main

What does this merge request do and why?

This adds support to evaluate with Gemini Pro 1.5.

Relates to #183 (closed)

How to set up and validate locally

  1. Ensure GCP environment variables are setup.

  2. Check out to this merge request's branch.

  3. Use the following config file, named it duochat_eval_config_gemini.json.

      {
      "beam_config": {
        "pipeline_options": {
          "runner": "DirectRunner",
          "project": "dev-ai-research-0e2f8974",
          "region": "us-central1",
          "temp_location": "gs://prompt-library/tmp/",
          "save_main_session": false
        }
      },
      "input_bq_table": "dev-ai-research-0e2f8974.duo_chat_external.experiment_code_generation__input_v1",
      "output_sinks": [
        {
          "type": "local",
          "path": "data/output",
          "prefix": "experiment"
        }
      ],
      "throttle_sec": 0.1,
      "batch_size": 10,
      "input_adapter": "mbpp",
      "eval_setup": {
        "answering_models": [
          {
            "name": "gemini-1.5-pro-preview-0215",
            "prompt_template_config": {
              "templates": [
                {
                  "name": "empty",
                  "template_path": "data/prompts/duo_chat/answering/empty.txt.example"
                }
              ]
            }
          }
        ],
        "metrics": [
          {
            "metric": "independent_llm_judge",
            "evaluating_models": [
              {
                "name": "text-bison@latest",
                "prompt_template_config": {
                  "templates": [
                    {
                      "name": "claude-2",
                      "template_path": "data/prompts/duo_chat/evaluating/claude-2.txt.example"
                    }
                  ]
                }
              }
            ]
          }
        ]
      }
    }
  4. Run the follow command to kick off the pipeline.

    poetry run promptlib duo-chat eval --test-run --sample-size 1 --config-file=data/config/duochat_eval_config_gemini.json 
  5. View the perfect response 🎉 experiment_20240307_143350__independent_llm_judge-00000-of-00001.csv

Merge request checklist

  • I've ran the affected pipeline(s) to validate that nothing is broken.
  • Tests added for new functionality. If not, please raise an issue to follow up.
  • Documentation added/updated, if needed.
Edited by Tan Le

Merge request reports