Skip to content

Exclude a pair of user and assistant role messages that encountered an error from the new conversation/prompt

Problem

From this Slack message https://gitlab.slack.com/archives/C06D5C70MD2/p1723722056121179

Hi team, I just had a call with @HossamHamdy . He told me that he gets A1001 error quite a lot. He could even demo it in the call. He made two calls at around 11:29 a.m. UTC. The first one from IDE and the second one from GitLab UI (both on .com). His GL user name is HossamHamdy. Since we now have this very concrete feedback with timestamp, would it be a good use of anyone’s of your time to dig into this? I suppose we still don’t have statistics to tell us which error is most common and how common, but we have Hossam’s observation that he gets A1001 quite a lot.

Some of the users always get error responses from Duo Chat. This is because abnormal data exists in chat history.

Workaround

Run /clean or /reset

Client side error

We see the error all messages must have non-empty content except for the optional final assistant message in AI Gateway. This is a client side error that GitLab-Sidekiq composes messages incorrectly hence Anthropic API returns an error:

{
    "status_code": 400,
    "exception_class": "AnthropicAPIStatusError",
    "backtrace": "Traceback (most recent call last):\n  File \"/home/shinya/gitlab-development-kit/gitlab-ai-gateway/ai_gateway/models/anthropic.py\", line 271, in generate\n    suggestion = await self.client.messages.create(\n  File \"/home/shinya/gitlab-development-kit/gitlab-ai-gateway/.venv/lib/python3.10/site-packages/anthropic/resources/messages.py\", line 1766, in create\n    return await self._post(\n  File \"/home/shinya/gitlab-development-kit/gitlab-ai-gateway/.venv/lib/python3.10/site-packages/anthropic/_base_client.py\", line 1815, in post\n    return await self.request(cast_to, opts, stream=stream, stream_cls=stream_cls)\n  File \"/home/shinya/gitlab-development-kit/gitlab-ai-gateway/.venv/lib/python3.10/site-packages/anthropic/_base_client.py\", line 1509, in request\n    return await self._request(\n  File \"/home/shinya/gitlab-development-kit/gitlab-ai-gateway/.venv/lib/python3.10/site-packages/anthropic/_base_client.py\", line 1610, in _request\n    raise self._make_status_error_from_response(err.response) from None\nanthropic.BadRequestError: Error code: 400 - {'type': 'error', 'error': {'type': 'invalid_request_error', 'message': 'messages.25: all messages must have non-empty content except for the optional final assistant message'}}\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"/home/shinya/gitlab-development-kit/gitlab-ai-gateway/ai_gateway/instrumentators/model_requests.py\", line 114, in watch\n    yield watcher\n  File \"/home/shinya/gitlab-development-kit/gitlab-ai-gateway/ai_gateway/models/anthropic.py\", line 278, in generate\n    raise AnthropicAPIStatusError.from_exception(ex)\nai_gateway.models.anthropic.AnthropicAPIStatusError: 400 Error code: 400 - {'type': 'error', 'error': {'type': 'invalid_request_error', 'message': 'messages.25: all messages must have non-empty content except for the optional final assistant message'}}\n",
    "correlation_id": "01J5CX537GZ671FWCZ3E326V3E",
    "extra": {
        "model_engine": "anthropic",
        "model_name": "claude-3-5-sonnet-20240620"
    },
    "logger": "exceptions",
    "level": "error",
    "type": "mlops",
    "stage": "main",
    "timestamp": "2024-08-16T06:14:19.298199Z",
    "message": "400 Error code: 400 - {'type': 'error', 'error': {'type': 'invalid_request_error', 'message': 'messages.25: all messages must have non-empty content except for the optional final assistant message'}}"
}

Steps to reproduce

This can be reproduced locally by setting an empty content a the chat message in storage.

  1. Change the GitLab-Rails codebase in the following: 
    diff --git a/ee/lib/gitlab/llm/graphql_subscription_response_service.rb b/ee/lib/gitlab/llm/graphql_subscription_response_service.rb
index 13906175d81f..ed9379a0ffd3 100644
--- a/ee/lib/gitlab/llm/graphql_subscription_response_service.rb
+++ b/ee/lib/gitlab/llm/graphql_subscription_response_service.rb
@@ -18,7 +18,7 @@ def response_message
          data[:role] ||= AiMessage::ROLE_ASSISTANT
          data.merge!(
            user: user,
-            content: response_modifier.response_body,
+            content: "",
            errors: response_modifier.errors,
            extras: response_modifier.extras,
            context: AiMessageContext.new(resource: resource)
  2. Send a Duo Chat request. Confirm that it's successful.
  3. Send a Duo Chat request again. Confirm that you get A1001 error.

2024-08-16_15-50

Live debug

For the full details (logs, etc), see the following link:

https://www.youtube.com/watch?v=L08x8eJmvsI (GitLab internal only)

Proposal

  • Investigate when we could set an empty assistant-role error message with an empty content. It's most likely we're failing an error handling.
  • Exclude a pair of user and assistant role messages that encountered an error from the new conversation/prompt.
Edited by Torsten Linz