Wrap the text palm model with custom API
This MR extends the Code Suggestions model gateway and wraps the Palm text model (text-bison-001) with a custom API endpoint:
/v2/generate/palm
To start the code suggestions server with the implemented endpoint only, please provide the GENERATIVE_AI_ONLY=True
env variable.
Note that to authenticate requests to VertexAi, we need to provide the path to a JSON key (GOOGLE_APPLICATION_CREDENTIALS
env variable) generated for a service account or a personal account following the hack with legacy credentials. Please, contact @achueshev for the hack explanation to prevent any security issues.
How to query the Plam model using Curl:
curl -v -X POST \
-H "Content-Type: application/json" \
"http://localhost:5052/v2/generate/palm" \
--data-binary @- << EOF
{
"prompt_version": 1,
"model": {
"name": "text-bison-001",
"content": "string",
"temperature": 0.2,
"max_decode_steps": 16,
"top_p": 0.95,
"top_k": 40
}
}
EOF
Output:
{
"id":"fb6e9b1bf86a470db83e3f89c2fce243",
"objective":"text",
"created":1682339580,
"model":{
"name":"text-bison-001",
"choices":[
{
"text":"\n\nThe hint is a function that takes a string as input and returns a",
"index":0
}
]
}
}
This MR targets the non-main branch to prevent any concerns with the Code Suggestions development.
Ref: https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist/-/issues/64 rED: https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist/-/issues/142