Skip to content

Remove custom end of stream marker for CS

Tan Le requested to merge remove-custom-eos-marker into main

What does this merge request do and why?

Remove custom end of stream marker for CS. Stream/chunk transfer termination is handled at the HTTP layer by FastAPI. We do need to explicitly send any extra marker.

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

  1. Check out to this merge request's branch.
  2. Ensure a local Docker image built successfully.
    docker buildx build --platform linux/amd64 \
      -t ai-gateway:dev .
  3. Run a local service on Docker.
    docker run --platform linux/amd64 --rm \
      -p 5052:5052 \
      -e AUTH_BYPASS_EXTERNAL=true \
      -v $PWD:/app -it ai-gateway:dev
  4. Execute a cURL request with streaming, noticed the stream terminate correctly without any custom marker.
    curl \
      --url http://codesuggestions.gdk.test:5052/v2/code/generations \
      --header 'Content-Type: application/json' \
      --header 'X-Gitlab-Authentication-Type: oidc' \
      --data '{
      "current_file": {
        "file_name": "math.py",
        "content_above_cursor": "# Hello world",
        "content_below_cursor": "\n"
      },
      "prompt_version": 2,
      "prompt": "write me a Dr Seus poem about GitLab AIDevSecOps for 30 words\n",
      "model_provider": "anthropic",
      "stream": true
    }
    '
    
    * Connected to codesuggestions.gdk.test (172.16.123.2) port 5052 (#0)
    > POST /v2/code/generations HTTP/1.1
    > Host: codesuggestions.gdk.test:5052
    > User-Agent: curl/8.1.2
    > Accept: */*
    > Content-Type: application/json
    > X-Gitlab-Authentication-Type: oidc
    > authorization: Bearer 
    > Content-Length: 384
    > 
    < HTTP/1.1 200 OK
    < date: Fri, 08 Dec 2023 06:55:33 GMT
    < server: uvicorn
    < content-type: text/event-stream; charset=utf-8
    < x-process-time: 1.3241759550001007
    < x-request-id: 2f50e852b2f84cb0838eb7abc2f1cc7c
    < transfer-encoding: chunked
    < 
    
    Here is a 30-word Dr. Seuss-style poem about GitLab AIDevSecOps:
    
    GitLab AI, GitLab AI, 
    Helps devs and secops play.
    With bots that code and bots that test,
    Efficiency is best!
    AI tools assist the team, 
    To build apps like a dream.
    With GitLab's automation power,
    Delivery takes less hour!* Connection #0 to host codesuggestions.gdk.test left intact

Merge request checklist

  • Tests added for new functionality. If not, please raise an issue to follow up.
  • Documentation added/updated, if needed.
Edited by Tan Le

Merge request reports