Skip to content

Fix prefix incorrectly parsed in Vertex request

Tan Le requested to merge fix-vertex-code-suggestion-request into main

What does this merge request do and why?

Fix prefix incorrectly parsed in Vertex request

We pass StructuredPrompt to all get_batch_completions calls but this is presumed string type in VertexCodeModelHandle. This results in 400 errors and subsequent empty response from Vertex code model such as code-gecko.

WARNING:root:Model name: code-gecko@002. Prediction failed for prompt: ('Human: func NewFormatService(containe
r container.Container) FormatService\nfunc (m *formatService) FindAllFormats() *[]model.Format\npackage service\n\nimport (\n\t"github.com/ybkuroki/go-webapp-sample/contain
er"\n\t"github.com/ybkuroki/go-webapp-sample/model"\n)\n\n// FormatService is a service for managing master data such as format and category.\ntype FormatService interface 
{\n\tFindAllFormats() *[]model.Format\n}\n\ntype formatService struct \n\nAssistant:\n', '', ''). Error: 400 Instance prefix cannot be empty..                              

How to set up and validate locally

To reproduce this bug, we can run a test on the code-suggestions eval with the following config.

{
  "beam_config": {
    "pipeline_options": {
      "runner": "DirectRunner",
      "project": "dev-ai-research-0e2f8974",
      "region": "us-central1",
      "temp_location": "gs://prompt-library/tmp/",
      "save_main_session": false
    }
  },
  "input_bq_table": "dev-ai-research-0e2f8974.code_suggestion.input_testcases_v1",
  "output_bq_table": "dev-ai-research-0e2f8974.code_suggestion_experiments.tle_test_run_go",
  "throttle_sec": 0.1,
  "min_length": 25,
  "batch_size": 32,
  "eval_setups": [
    {
      "llm_config": {
        "name": "code-gecko@002"
      },
      "languages": ["go"],
      "prompt_template_config": {
        "templates": [
          {
            "name": "example_template",
            "template_path": "data/prompts/code_suggestions_simple.txt.example"
          }
        ]
      },
      "transformations": ["function_signatures", "imports"],
      "post_transformations": [],
      "include_suffix": true
    }
  ]
}
poetry run promptlib code-suggestions eval --test-run --sample-size 1 --config-file data/config/codesuggestion_config.json

Merge request checklist

  • I've ran the affected pipeline(s) to validate that nothing is broken.
  • Tests added for new functionality. If not, please raise an issue to follow up.
  • Documentation added/updated, if needed.
Edited by Tan Le

Merge request reports