Gracefully handle null content in Duo Chat history

What I see:

Requests to /v1/chat/agent where there is content: null anywhere in the history will return a 422.

Example

What I expect to see:

Requests to /v1/chat/agent where there is content: null anywhere in the history will return a helpful response.

More context

If Duo Chat ever returns null for an assistant message in the Chat history, sending that history with a new Duo Chat request (which is what we do unless /clear is run) will result in a 422.

The AI Gateway should handle this more gracefully and not return a 422. We want to keep validations in place for user messages, but maybe we can relax this requirement for assistant responses?

Example request body for AI Gateway API request that returns a 422 (tested at http://127.0.0.1:5052/docs#/chat/chat_v1_chat__chat_invokable__post):

{
  "prompt_components": [
    {
      "type": "string",
      "metadata": {
        "source": "string",
        "version": "string"
      },
      "payload": {
        "content": [{"role":"user","content":"How do I change my password in GitLab?"},{"role":"assistant","content":null},{"role":"user","content":"hi!"}],
        "provider": "anthropic",
        "model": "claude-2.0",
        "params": {
          "stop_sequences": [
            "\n\nHuman",
            "Observation:"
          ],
          "temperature": 0.2,
          "max_tokens_to_sample": 2048
        },
        "model_endpoint": "string",
        "model_api_key": "string"
      }
    }
  ],
  "stream": false
}

added Category:Duo Chat bugfunctional typebug labels

added devopsai-powered groupduo chat sectiondata-science labels

cc @juan-silva

This is a bit of an edge case but popped up for a team member using Duo Chat

@jessieay I wonder if we just should not add such responses to the chat history? They are not helpful and won't add any additional context so LLM can answer the question better. WDYT?

@brytannia agree that they are not helpful.

I believe that the JSON body sent to Anthropic messages API must alternate between user and assistant messages or we will get a validation error.

So, if we want to remove the null assistant response we would also need to remove the associated user question from the chat history. That could be a good plan.

Or, we could convert the null assistant message content into an empty string. I think that the AI Gateway API would accept this, not sure about the Anthropic API.

Just confirmed that the following request body to AI Gateway /v1/chat/agent:

{
  "prompt_components": [
    {
      "type": "string",
      "metadata": {
        "source": "string",
        "version": "string"
      },
      "payload": {
        "content": [ {"role":"user","content":"How do I change my password in GitLab?"},{"role":"user","content":"hi!"}],
        "provider": "anthropic",
        "model": "claude-2.0",
        "params": {
          "stop_sequences": [
            "\n\nHuman",
            "Observation:"
          ],
          "temperature": 0.2,
          "max_tokens_to_sample": 2048
        },
        "model_endpoint": "string",
        "model_api_key": "string"
      }
    }
  ],
  "stream": false
}

Yields an Anthropic API error because we are including 2 user messages in a row rather than separating them by an assistant message.

cc @brytannia

@jessieay makes sense!

@lesley-r I remember that you added some change (to ZeroShot executor probably) that suppose to prevent storing the same type of message in a row (that was a fix related to the Sidekiq issue). I can't find it in the code for some reason, maybe you know what I'm talking about and could find a link?

@brytannia @jessieay

Here is the code for deduplicating the roles

https://gitlab.com/gitlab-org/gitlab/blob/a4fc4e6c0711ced651c7683220d2fba5a91adac7/ee/lib/gitlab/llm/chain/agents/zero_shot/prompts/anthropic.rb#L40

thanks @lesley-r! I'll port it to the SingleAction executor (it supposed to be a follow-up) so we have this behaviour in v2 chat as well. Then we can reject null answers and it should work all together!

@brytannia This was fixed in the monolith via !163371 (merged) (thanks @lesley-r !)

Do you want to create a separate issue for fixing this in the AI Gateway or should we re-assign this issue to you now?

@jessieay @lesley-r for v2 chat agent we need to fix it also in monolith but a bit in different place: here it's used in SingleActionExecutor (deduplication function need to be ported as well ) So I guess this issue will be fine.

However, I don't have capacity to tackle this issue right now. I will talk to Juan before I go on PTO next week about who can pick it up.

marked this issue as related to #479460 (closed)

mentioned in merge request !163371 (merged)

assigned to @lesley-r

changed milestone to %17.4

@lesley-r since you are already tackling this via !163371 (merged) I've assigned this issue to you

assigned to @brytannia

mentioned in issue #466910 (closed)

mentioned in merge request !163781 (merged)

added workflowin review label

mentioned in epic gitlab-org#13533 (closed)

@lesley-r should we use this issue to track the sentry error you saw last week or are we tracking that in another issue?

The associated log line:

400 Error code: 400 - {'type': 'error', 'error': {'type': 'invalid_request_error', 'message': 'messages: roles must alternate between "user" and "assistant", but found multiple "user" roles in a row'}}

@jessieay

Created a new issue for it!

@jessieay how did you extracted the associated log? I don't see it in the sentry issue page and when I filter logs by correlation_id I don't see anything either. Only error that AI gateway not reachable. Could you help me find what I'm missing?

Hi @brytannia !

The steps I followed are somewhat covered in the Duo Chat Runbook.

I will list them here, and please add anything you find helpful to the Runbook:

To find root cause of A1001 (which is the error code Lesley reported in Slack):

1. Search for json.error_code : "A1001" in Sidekiq logs (json.subcomponent : "llm" and json.error_code : "A1001"). Example
2. Find top result and grab correlation_id. Search AI Gateway logs for the logs with the same one: json.jsonPayload.correlation_id : "CORRELATION_ID". Example
3. If you inspect the json.jsonPayload.backtrace of the AI Gateway logs you will see the end of the backtrace contains the following:

raise AnthropicAPIStatusError.from_exception(ex)
ai_gateway.models.anthropic.AnthropicAPIStatusError: 400 Error code: 400 - {'type': 'error', 'error': {'type': 'invalid_request_error', 'message': 'messages: roles must alternate between "user" and "assistant", but found multiple "user" roles in a row'}}

cc @lesley-r

And to clarify: I didn't use the Sentry error at all here because Lesley mentioned the error code I started with that.

@jessieay thanks for the information! Do I understand correctly, that you didn't use information from sentry error that you linked in the first comment?

@brytannia yes, sorry for the confusion. The Slack conversation began with a link to the Sentry error and then @lesley-r said:

It seems it is another cause of A1001 errors

So in my investigation I dug right into the A1001 errors rather than looking at Sentry.

cloned to #480928 (closed)

@lesley-r the fix for V2 was merged, as we don't have a stable way to reproduce this bug in production, and v2 isn't used wildly now, I'm going to close this bug as completed.

added workflowcomplete label and removed workflowin review label

closed

added AI label