Skip to content

Return empty list of choices on empty suggestion

Tan Le requested to merge 303-return-empty-list-on-empty-suggestions into main

What does this merge request do and why?

This ensures we return an empty list of choices when the model return an empty suggestion or if the score is below the confidence threshold. This convers both code completions and code generations.

How to set up and validate locally

  1. Check out to this merge request's branch.
  2. Patch the following file
    diff --git ai_gateway/models/fake.py ai_gateway/models/fake.py
    index 4dac4df..97d893c 100644
    --- ai_gateway/models/fake.py
    +++ ai_gateway/models/fake.py
    @@ -21,4 +21,4 @@ class FakePalmTextGenModel(TextGenBaseModel):
             top_p: float = 0.0,
             top_k: int = 0,
         ) -> Optional[TextGenModelOutput]:
    -        return TextGenModelOutput(text="fake code suggestion from PaLM Text", score=0)
    +        return TextGenModelOutput(text="", score=0)
  3. Ensure a local Docker image built successfully.
    docker buildx build --platform linux/amd64 \
      -t model-gateway:test .
  4. Run a local service on Docker.
    docker run -it --platform linux/amd64 --rm -p 5999:5000 \
      -e AUTH_BYPASS_EXTERNAL=true \
      -e USE_FAKE_MODELS=true \n
      -v $PWD:/app model-gateway:test
  5. Send a cURL request to the /v2/completions and /v2/code/generations endpoint
    $ curl --request POST \
      --url http://codesuggestions.gdk.test:5999/v2/completions \
      --header 'Content-Type: application/json' \
      --header 'X-Gitlab-Authentication-Type: oidc' \
      --header 'authorization: Bearer jwt \
      --data '{
      "prompt_version": 1,
      "project_path": "gitlab-org/gitlab",
      "project_id": 278964,
      "current_file": {
        "file_name": "main.js",
        "content_above_cursor": "// a binary search function ",
        "content_below_cursor": "\n"
      }
    }'
  6. Confirm the response show an empty choices list
    {
      "id": "id",
      "model": {
        "engine": "fake-palm-engine",
        "name": "fake-palm-model",
        "lang": "js"
      },
      "experiments": [
        {
          "name": "exp_truncate_suffix",
          "variant": 0
        }
      ],
      "object": "text_completion",
      "created": 1695211452,
      "choices": []
    }

Merge request checklist

  • Tests added for new functionality. If not, please raise an issue to follow up.
  • Documentation added/updated, if needed.

Closes #303 (closed)

Edited by Tan Le

Merge request reports