Merge custom llm_config.parameters with the default model parameters
What does this merge request do and why?
This fixes a confusing behaviour that happens when specifying custom llm_config.parameters
in the config file.
Custom parameters should replace default parameters if they happen to be already specified, but not replace the whole dict of default parameters.
The following config causes the stop
parameters to have no effect. This happens because the custom dict of parameters will replace the model specific dict of default parameters. In the case of code-gemma-2b
, stop
will only work if raw_response: true
is also specified. Thus if one wants to customize any of the model parameters, he/she will have to also replicate the default_parameters hardcoded in promptlib, which is a surprising behaviour.
"llm_config": {
"name": "code-gemma-2b",
"parameters": {
"raw_response": false,
"stop": [
"<|fim_prefix|>",
"<|fim_middle|>",
"<|fim_suffix|>",
"<|file_separator|>"
]
}
},
How to set up and validate locally
- Run the following pipeline with the configs below:
promptlib code-suggestions eval --config-file=data/config/code-completion/code_gemma_config.json --test-run --sample-size=1
This will work (when checking the model_completion column in the resulting table, one won't find any of the stop
strings, which means that they had an effect in the generated output of the model:
"llm_config": {
"name": "code-gemma-2b",
"parameters": {
"raw_response": true,
"stop": [
"<|fim_prefix|>",
"<|fim_middle|>",
"<|fim_suffix|>",
"<|file_separator|>"
]
}
},
This won't work (you will see the stop
strings in the model output):
"llm_config": {
"name": "code-gemma-2b",
"parameters": {
"stop": [
"<|fim_prefix|>",
"<|fim_middle|>",
"<|fim_suffix|>",
"<|file_separator|>"
]
}
},
Merge request checklist
-
I've ran the affected pipeline(s) to validate that nothing is broken. -
Tests added for new functionality. If not, please raise an issue to follow up. -
Documentation added/updated, if needed.