Skip to content

Wrap the text palm model with custom API

Alexander Chueshev requested to merge integrate_palm_api into internal-model-gateway

This MR extends the Code Suggestions model gateway and wraps the Palm text model (text-bison-001) with a custom API endpoint:

  • /v2/generate/palm

To start the code suggestions server with the implemented endpoint only, please provide the GENERATIVE_AI_ONLY=True env variable. Note that to authenticate requests to VertexAi, we need to provide the path to a JSON key (GOOGLE_APPLICATION_CREDENTIALS env variable) generated for a service account or a personal account following the hack with legacy credentials. Please, contact @achueshev for the hack explanation to prevent any security issues.

How to query the Plam model using Curl:

curl -v -X POST \
  -H "Content-Type: application/json" \
  "http://localhost:5052/v2/generate/palm" \
  --data-binary @- << EOF
{
  "prompt_version": 1,
  "model": {
    "name": "text-bison-001",
    "content": "string",
    "temperature": 0.2,
    "max_decode_steps": 16,
    "top_p": 0.95,
    "top_k": 40
  }
}
EOF

Output:

{
  "id":"fb6e9b1bf86a470db83e3f89c2fce243",
  "objective":"text",
  "created":1682339580,
  "model":{
    "name":"text-bison-001",
    "choices":[
        {
          "text":"\n\nThe hint is a function that takes a string as input and returns a",
          "index":0
        }
    ]
  }
}

This MR targets the non-main branch to prevent any concerns with the Code Suggestions development.

Ref: https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist/-/issues/64 rED: https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist/-/issues/142

Edited by Mon Ray

Merge request reports