Duo Workflow checkpoint write fails with EncodingError when logging a tolerated __start__ blob decode error from an old AI Gateway

Summary

Ai::DuoWorkflows::CreateCheckpointService#start_channel_from_blobs rescues decode errors on purpose. The rescue calls Gitlab::ErrorTracking.track_exception and returns {}. That log call raises a second exception: EncodingError: Invalid Unicode [...] at 94. Nothing catches the second exception. A fallback path that the code treats as harmless becomes a 500 response.

The trigger is an AI Gateway version that predates the incremental-checkpoints (channel blob) feature. That gateway sends blob data in a format the service does not expect. The decoded bytes are not valid UTF-8. The exception payload carries those bytes into Gitlab::JsonLogger, and Gitlab::Json.dump cannot serialize them.

This is a compatibility break between an old AI Gateway and current GitLab. It affects any self-managed or Duo Self-Hosted customer on an older gateway.

How to reproduce

  1. Run a current GitLab instance against an AI Gateway that predates the incremental-checkpoints feature. The stable-19-1-ee branch of ai-assist is one such snapshot.
  2. Start a Duo Chat conversation that runs through a workflow.
  3. Wait for the gateway to write its first checkpoint.

The feature-tests downstream pipeline reproduces this on every run. MR !6915 backports an unrelated one-line fix to stable-19-1-ee. Every merge train pipeline for that MR fails the same way, across multiple days.

Example: job 16559399993 in pipeline 2857562267. The failing spec is ee/spec/features/duo_chat/user_use_same_thread_across_projects_spec.rb. The assertion expect(page).to have_css('.duo-chat-message-complete', count: 4) never passes. The spec also reports a BrowserConsoleError for the 500 responses.

What happens

log/exceptions_json.log for a single job shows this sequence. Each entry has the same EncodingError: Invalid Unicode [...] at 94 signature, with different bytes each time.

  1. POST /api/:version/ai/duo_workflows/workflows/:id/checkpoints fails. This is the first checkpoint write from the old gateway.
  2. GraphqlController#execute:getWorkflowLatestCheckpoint fails on every poll from the frontend.
  3. GraphqlController#execute fails for the Duo Chat message mutation.

The GraphQL requests return 500. The browser console shows repeated Failed to load resource: the server responded with a status of 500 for /api/graphql. The chat conversation does not complete.

The relevant code is in ee/app/services/ai/duo_workflows/create_checkpoint_service.rb, around lines 125-134:

rescue StandardError => e
  # Runs after the transaction committed the header and blobs, so nothing here
  # may fail the write: a bad `__start__` costs the goal backfill only.
  ::Gitlab::ErrorTracking.track_exception(e, workflow_id: @workflow.id)
  {}

The comment above the method says the payload is "the fallback for a gateway old enough to send no blobs". The design intent is clear. The implementation does not match it.

What should happen

The rescue block logs the decode error and returns {}. The checkpoint write succeeds. The workflow continues with the goal backfill skipped. Duo Chat completes against an old AI Gateway.

Additional context

Two possible fix directions:

  • Gitlab::ErrorTracking.track_exception, or Gitlab::JsonLogger and Gitlab::Json.dump under it, could handle exception payloads that contain non-UTF-8 bytes. A tolerated error should not raise a second, unhandled exception on the log path.
  • start_channel_from_blobs could avoid passing raw decoded blob bytes into anything bound for structured logging. For example, it could log only the blob length or a hex digest.

Both changes could apply together. The first protects other callers with the same problem. The second contains the fix to this service.

Links: