Duo Workflow fails to create workflow

Issue Summary

On March 28, 2025, Duo-Workflow stopped functioning in production due to a header case sensitivity issue. The workflow executor began failing with authentication errors when processing gRPC connections.

Customer Impact: Private beta participants not being able to create new workflows

Error Details

2025-03-31T16:53:06.586+0900	ERROR	duo-workflow-executor/main.go:113	Failed to run	{"correlation_id": "01JQNJY53A5JDD416B1QZ6MGN7", "workflow_id": "1005965", "error": "failed to execute runner: error receiving event: rpc error: code = Unauthenticated desc = Header mismatch 'X-Gitlab-Realm'"}
main.main
	/builds/gitlab-org/duo-workflow/duo-workflow-executor/main.go:113
runtime.main
	/usr/local/go/src/runtime/proc.go:283

Timeline

Deploy of the problematic change: 31st March 2025 - 3:05am UTC (MR)

First error report: 31st March 2025 - 7:53am UTC

Merge of the fix: 31st March 2025 - 12:43pm

Fix deployed to Production: 31st March 2025 - 9:12pm

Root Cause

A recent merge request changed the case of HTTP headers from their original capitalized format to lowercase. This created an incompatibility with the Language Server, which specifically expects headers to be capitalized (e.g., X-Gitlab-Realm rather than x-gitlab-realm).

Temporary Resolution

The issue was temporarily addressed by reverting the problematic change: gitlab-org/gitlab!186407 (merged)

Revert Details

1. Duo Workflow was failing with "failed to execute runner: error receiving event: rpc error: code = Unauthenticated desc = Header mismatch 'X-Gitlab-Realm'"
2. Language Server expects headers to be capitalized, but the recent change converted them to lowercase
3. A permanent fix will need to be implemented in the Language Server, but this revert serves as a temporary measure to restore production functionality

Follow up discussion in gitlab-org/gitlab#531521