Duo Agent Platform - Requests being rejected due to gRPC header size limit

Summary

A number of workhorse requests seem to be rejected by Duo Workflow Service gRPC as the headers / metadata of the request are too big.

Workhorse logs

Between 2025-08-29 and 2025-09-12 there were 42 entries.

These seem to be either exceeding the soft limit of 8192B or the hard limit of 16384B.

Frequently the largest header is x-gitlab-feature-enabled-by-namespace-ids.

May be linked to JWT generation, with logs commonly only consisting of a JWKS refreshed event then Finished ExecuteWorkflow RPC event with no workflow_id.

Example log

Example error JSON

{
  "correlation_id": "01K4T6T67PCYSVQTW9EYWHWZVZ",
  "environment": "gprd",
  "error": "duoworkflow: failed to read a gRPC message: rpc error: code = ResourceExhausted desc = received metadata size exceeds hard limit (17474 vs. 16384);  :path:65B :authority:58B :method:43B :scheme:43B content-type:60B te:42B grpc-accept-encoding:66B user-agent:56B cf-ray:58B cdn-loop:59B accept-encoding:55B x-gitlab-feature-enabled-by-namespace-ids:12665B x-gitlab-unidirectional-streaming:72B cf-ipcountry:46B cf-visitor:60B cf-connecting-ip:59B x-gitlab-instance-id:88B x-gitlab-client-name:71B x-gitlab-version:54B x-gitlab-global-user-id:99B x-gitlab-oauth-token:116B x-gitlab-correlation-id:81B x-gitlab-feature-enablement-type:68B x-gitlab-authentication-type:64B x-gitlab-enabled-feature-flags:116B x-gitlab-enabled-instance-verbose-ai-logs:73B x-gitlab-host-name:60B x-gitlab-realm:50B x-gitlab-user-id:56B via:45B x-forwarded-for:88B x-forwarded-proto:54B forwarded:72B x-cloud-trace-context:106B authorization:2616B",
  "level": "error",
  "logtag": "F",
  "method": "GET",
  "msg": "",
  "shard": "default",
  "stage": "main",
  "tag": "ai-assisted-workhorse.var.log.containers.gitlab-webservice-ai-assisted-6df89fbf76-8k6d8_gitlab_gitlab-workhorse-333018d937c41d408ea969f35e3c4dd878383eca52f98b2dd9dbf4de1f956f84.log",
  "tier": "sv",
  "time": "2025-09-10T16:19:49.000Z",
  "type": "ai-assisted",
  "uri": "/api/v4/ai/duo_workflows/ws",
}