<new_code></new_code> tags not being removed in Duo code suggestion/completion response to vscode with self-hosted llama model

Summary

Code suggestion/completion responses include <new_code></new_code> tags. e.g.

// add two numbers
<new_code>int result = 0; result = 5 + 7;</new_code>

for <new_code>(int i = 0; i < 10; i++)</new_code>

Environment:

  • GitLab version 18.3.1
  • vscode version 1.103.2
  • GitLab Workflow extension version 6.40.1
  • AIGW image gitlab/model-gateway:self-hosted-v18.3.0-ee
  • LLM RedHatAI/Llama-3.3-70B-Instruct-FP8-dynamic

Steps to reproduce

  1. Deploy vLLM instance running RedHatAI/Llama-3.3-70B-Instruct-FP8-dynamic model.
  2. Deploy AIGW and configure GitLab Duo to use self-hosted model.
  3. In vscode (or the Web IDE) create an empty test.c file and add the comment line // add two numbers
  4. Observe the tags in the suggested code.
  5. Start a new line with the text for
  6. Observe the tags in the suggested code.

Example Project

What is the current bug behavior?

Tags used to denote new code suggestion from model are not removed when the code is presented in the IDE.

What is the expected correct behavior?

Tags should be removed from the result.

Relevant logs and/or screenshots

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info 
 (For installations with omnibus-gitlab package run and paste the output of: \`sudo gitlab-rake gitlab:env:info\`)  (For installations from source run and paste the output of: \`sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production\`)

Results of GitLab application Check

Expand for output related to the GitLab application check 
(For installations with omnibus-gitlab package run and paste the output of: `sudo gitlab-rake gitlab:check SANITIZE=true`)

(For installations from source run and paste the output of: `sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true`)

(we will only investigate if the tests are passing)

Possible fixes

Patch release information for backports

If the bug fix needs to be backported in a patch release to a version under the maintenance policy, please follow the steps on the patch release runbook for GitLab engineers.

Refer to the internal "Release Information" dashboard for information about the next patch release, including the targeted versions, expected release date, and current status.

High-severity bug remediation

To remediate high-severity issues requiring an internal release for single-tenant SaaS instances, refer to the internal release process for engineers.

Edited by 🤖 GitLab Bot 🤖