Switch to using RESOURCE_EXHAUSTED for concurrency limit errors
In our concurrency limiter, when we hit hard limits in terms of queue wait time and queue size, we emit a grpc UNAVAILABLE
error code. Originally this was chosen because of the gRPC code description:
The service is currently unavailable. This is most likely a transient condition, which can be corrected by retrying with a backoff. Note that it is not always safe to retry non-idempotent operations.
However, after discussion in https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/16844#note_1175622086, it turns out this is not a great error code since it indicates something is wrong with the system. Instead, using RESOURCE_EXHAUSTED
is more in line with the failure mode, since we don't want the caller to retry at this point if we are experiencing concurrency queue limits.
Switch from Unavailable
to ResourceExhausted