Skip to content

Merge custom llm_config.parameters with the default model parameters

Bruno Cardoso requested to merge bc/merge-custom-parameters into main

What does this merge request do and why?

This fixes a confusing behaviour that happens when specifying custom llm_config.parameters in the config file. Custom parameters should replace default parameters if they happen to be already specified, but not replace the whole dict of default parameters.

The following config causes the stop parameters to have no effect. This happens because the custom dict of parameters will replace the model specific dict of default parameters. In the case of code-gemma-2b, stop will only work if raw_response: true is also specified. Thus if one wants to customize any of the model parameters, he/she will have to also replicate the default_parameters hardcoded in promptlib, which is a surprising behaviour.

"llm_config": {
  "name": "code-gemma-2b",
  "parameters": {
      "raw_response": false,
      "stop": [
          "<|fim_prefix|>",
          "<|fim_middle|>",
          "<|fim_suffix|>",
          "<|file_separator|>"
      ]
  }
},

How to set up and validate locally

  1. Run the following pipeline with the configs below:
promptlib code-suggestions eval --config-file=data/config/code-completion/code_gemma_config.json --test-run --sample-size=1

This will work (when checking the model_completion column in the resulting table, one won't find any of the stop strings, which means that they had an effect in the generated output of the model:

"llm_config": {
  "name": "code-gemma-2b",
  "parameters": {
      "raw_response": true,
      "stop": [
          "<|fim_prefix|>",
          "<|fim_middle|>",
          "<|fim_suffix|>",
          "<|file_separator|>"
      ]
  }
},

This won't work (you will see the stop strings in the model output):

"llm_config": {
  "name": "code-gemma-2b",
  "parameters": {
      "stop": [
          "<|fim_prefix|>",
          "<|fim_middle|>",
          "<|fim_suffix|>",
          "<|file_separator|>"
      ]
  }
},

Merge request checklist

  • I've ran the affected pipeline(s) to validate that nothing is broken.
  • Tests added for new functionality. If not, please raise an issue to follow up.
  • Documentation added/updated, if needed.

Merge request reports