Draft: Code suggestions streaming experiment

What does this MR do and why?

Info: The experiment, not intended to be merged

An experiment of adding a streaming option to /code_suggestions/completions API endpoint

Screenshots or screen recordings

completions_streaming_poc

As you can see in the video, the API endpoint streams responses in batches instead of streaming every event. I'm still trying to figure out why this is happening. It could be related to how Workhorse operates.

How to set up and validate locally

1. Copy the following script to a .js file

async function fetchDataFromStream() {
  const apiToken = 'YOUR_API_TOKEN';
  const apiUrl = 'http://127.0.0.1:3000/api/v4/code_suggestions/completions';


  try {
    const stream = true
    const rawResponse = false
    const contentAboveCursor = `package main

// Generate the code to create a web server
// that uses a single home handler with an index page
// that displays Hello world on the page
`
    const response = await fetch(apiUrl, {
      method: 'POST',
      headers: {
        'Content-Type': 'application/json', // Set the appropriate content type
        'Private-Token': apiToken, // Use 'Private-Token' for GitLab PAT
      },
      body: JSON.stringify({
        'current_file': {
          'file_name': 'main.go',
          'content_above_cursor': contentAboveCursor,
          'content_below_cursor': '\n}'
        },
        'stream': stream
      }),
    });

    if (!response.ok) {
      throw new Error(`HTTP error! Status: ${response.status}`);
    }

    const reader = response.body.getReader();

    while (true) {
      const { done, value } = await reader.read();

      if (done) {
        console.log('Streaming response is complete.');
        break;
      }

      const rawContent = new TextDecoder().decode(value);
      if (rawResponse) {
        console.log(rawContent);
      } else {
        // Split the content into lines
        const lines = rawContent.split('\n');

        // Initialize an array to store data objects
        // const dataObjects = [];

        // Iterate through each line
        for (let i = 0; i < lines.length; i++) {
          // Check if the line contains "data:"
          if (lines[i].includes("data:")) {
            // Find the next line with the JSON data
            const jsonData = lines[i].replace('data: ', '');

            // Check if the jsonData is not empty
            if (jsonData) {
              // Parse the JSON data
              const dataObject = JSON.parse(jsonData);
              // dataObjects.push(dataObject);

              if (dataObject.completion !== undefined) {
                // console.log(dataObject.completion);
                process.stdout.write(`${dataObject.completion}`);
              }
            }
          } else if (!stream) {
              const jsonData = rawContent
              const dataObject = JSON.parse(jsonData);

              if (dataObject.completion !== undefined) {
                process.stdout.write(`${dataObject.completion}`);
              }
          }
        }
      }
      // You can process or display the streamed data as needed

    }
  } catch (error) {
    console.error('Error:', error);
  }
}

fetchDataFromStream();

2. Provide your local GitLab Token
3. Run node filename.js

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

• I have evaluated the MR acceptance checklist for this MR.