Skip to content

Do not render chunks once full message is received

Nicolas Dular requested to merge nd/stream-order-issue into master

What does this MR do and why?

We do not guarantee the order when subscription messages are received. This is the case for chunks, but also for the full message.

So it could happen that we receive the full message from aiCompletion but then still receive one or more chunks. In this case we should stop trying to compute the full message based on the received chunks and ignore the chunk. Otherwise this would lead to a bug where the user only sees the last chunk rendered.

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

Before After
before after

How to set up and validate locally

This is hard to replicate as it fully relies on the order of the streamed messages that is not guaranteed. With the following script, you can replicate it:

  1. Use a fixed clientSubscriptionId to send the message to
diff --git a/ee/app/assets/javascripts/ai/tanuki_bot/components/app.vue b/ee/app/assets/javascripts/ai/tanuki_bot/components/app.vue
index 59ad49a90c11..ceed972d4e24 100644
--- a/ee/app/assets/javascripts/ai/tanuki_bot/components/app.vue
+++ b/ee/app/assets/javascripts/ai/tanuki_bot/components/app.vue
@@ -76,7 +76,7 @@ export default {
           return {
             userId: this.userId,
             resourceId: this.resourceId || this.userId,
-            clientSubscriptionId: this.clientSubscriptionId,
+            clientSubscriptionId: '123',
             htmlResponse: false,
           };
         },
diff --git a
  1. Run the following script to get the last chunk out of order
chunk1 = Gitlab::Llm::AiMessage.for(action: :chat).new({user: User.first, resource: User.first, content: "hello", role: Gitlab::Llm::AiMessage::ROLE_ASSISTANT, client_subscription_id: '123', request_id: 'foobar', chunk_id: 1})

chunk2 = Gitlab::Llm::AiMessage.for(action: :chat).new({user: User.first, resource: User.first, content: " world", role: Gitlab::Llm::AiMessage::ROLE_ASSISTANT, client_subscription_id: '123', request_id: 'foobar', chunk_id: 2})

full_message = Gitlab::Llm::AiMessage.for(action: :chat).new({user: User.first, resource: User.first, content: "hello world.", role: Gitlab::Llm::AiMessage::ROLE_ASSISTANT, client_subscription_id: '123', request_id: 'foobar', chunk_id: nil})

chunk3 = Gitlab::Llm::AiMessage.for(action: :chat).new({user: User.first, resource: User.first, content: ".", role: Gitlab::Llm::AiMessage::ROLE_ASSISTANT, client_subscription_id: '123', request_id: 'foobar', chunk_id: 3})


GraphqlTriggers.ai_completion_response(chunk1)
GraphqlTriggers.ai_completion_response(chunk2)
GraphqlTriggers.ai_completion_response(full_message)
GraphqlTriggers.ai_completion_response(chunk3)

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Nicolas Dular

Merge request reports