DWS: Increase metadata soft limit across all DWS gRPC clients

Overview

Currently, a flow may fail due to the following error:

duoworkflow: failed to read a gRPC message: rpc error: code = ResourceExhausted desc = received metadata size exceeds soft limit (8689 vs. 8192) ...

Full error

duoworkflow: failed to read a gRPC message: rpc error: code = ResourceExhausted desc = received metadata size exceeds soft limit (8689 vs. 8192); :path:65B :authority:76B :method:43B :scheme:43B content-type:60B te:42B grpc-accept-encoding:66B user-agent:56B traceparent:98B x-gitlab-root-namespace-id:66B x-gitlab-version:54B x-gitlab-agent-platform-model-metadata:172B x-gitlab-feature-enablement-type:68B x-gitlab-enabled-feature-flags:87B x-gitlab-host-name:60B x-gitlab-namespace-id:61B x-gitlab-unidirectional-streaming:72B x-gitlab-project-id:51B x-gitlab-oauth-token:116B x-gitlab-user-id:56B x-gitlab-global-user-id:99B x-gitlab-feature-enabled-by-namespace-ids:3730B x-gitlab-enabled-instance-verbose-ai-logs:73B x-gitlab-instance-id:88B x-gitlab-realm:50B x-gitlab-authentication-type:64B x-gitlab-language-server-version:70B x-gitlab-correlation-id:75B x-gitlab-client-name:71B via:45B x-forwarded-for:73B x-forwarded-proto:54B forwarded:70B authorization:2616B x-cloud-trace-context:109B

Logs: https://log.gprd.gitlab.net/app/r/s/vjItk

It happens because the headers/metadata that we send to DWS contain a lot of data (more than 8KB). According to the breakdown, x-gitlab-feature-enabled-by-namespace-ids is usually large and the header is used for internal tracking.

Proposal

We could consider optimizing the metadata, but a more reliable approach would be to increase the soft limit across all gRPC clients (Workhorse, Node/Golang Executor, DWS server limit).

It's 8KB by default, we can consider 50KB or 100KB. Considering that the full message limit is 4MB, it should be fine.

The fields that need updating are:

• grpc.max_metadata_size
• grpc.absolute_max_metadata_size

Updating grpc.max_metadata_size and grpc.absolute_max_metadata_size is sufficient because grpc.max_metadata_size has the following description:

Maximum metadata size (soft limit), in bytes. Note this limit applies to the
   max sum of all metadata key-value entries in a batch of headers. Some random
   sample of requests between this limit and
   `GRPC_ARG_ABSOLUTE_MAX_METADATA_SIZE` will be rejected. Defaults to maximum
   of 8 KB and `GRPC_ARG_ABSOLUTE_MAX_METADATA_SIZE` * 0.8 (if set).

Which probably means that we have a larger number of requests with large metadata, just a small portion of them fails randomly

Related to #569963 (closed)