Skip to content

Add `stream: true` to /code_suggestions/completions API

Vitali Tatarintev requested to merge ck3g-add-streaming-to-code_completions-api-endpoint into master

What does this MR do and why?

Related to #429366 (closed)

Add stream option to /code_suggestions/completions API endpoint.

Screenshots or screen recordings

Demo on GitLab Unfiltered https://www.youtube.com/watch?v=PY5n44-ndAY

Related articles

How to use the http.ResponseController type

Dependencies

Open MRs required for the change:

How to set up and validate locally

  1. Fetch a Model Gateway branch to your localhost gitlab-org/modelops/applied-ml/code-suggestions/ai-assist!435 (merged)
  2. Follow instructions to configure and run Model Gateway on localhost https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist#how-to-run-the-server-locally
  3. In the gdk directory `export CODE_SUGGESTIONS_BASE_URL='http://0.0.0.0:5052'
  4. Compile Workhorse cd workhorse; make
  5. Run gdk start
  6. Use one of the commands below to call APIs

localhost <-> Model Gateway <-> Anthropic

curl --request POST \
        --url http://0.0.0.0:5052/v2/code/completions \
        --header 'Content-Type: application/json' \
        --header 'X-Gitlab-Authentication-Type: oidc' \
        --data '{
    "current_file": {
      "file_name": "calc.go",
      "content_above_cursor": "// Generate the code to create a new calculator type\n// the type should have the following methods:\n// add, sub, multiply, divide, sqrt, abs, sin, cos",
      "content_below_cursor": "\n"
    },
    "prompt_version": 2,
    "prompt": "Generate the code in Golang to create a new calculator type\nthe type should have the following methods:\nadd, sub, multiply, divide, sqrt, abs, sin, cos",
    "model_provider": "anthropic",
    "stream": true
  }
  ' --no-buffer

localhost <-> GitLab API (Workhorse) <-> Model Gateway <-> Anthropic

curl --request POST \
        --url http://127.0.0.1:3000/api/v4/code_suggestions/completions \
        --header 'Content-Type: application/json' \
        --header 'Private-Token: YOUR_LOCAL_API_TOKEN' \
        --data '{
    "current_file": {
      "file_name": "calc.go",
      "content_above_cursor": "// Generate the code to create a new calculator type\n// the type should have the following methods:\n// add, sub, multiply, divide, sqrt, abs, sin, cos",
      "content_below_cursor": "\n"
    },
    "stream": true
  }
  ' --no-buffer

This chain currently does not work. I'm trying to figure out why.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Matt Nohr

Merge request reports