gRPC error spikes

Rails occasionally returns 502 and 503 HTTP status codes for various APIs kas is calling. That results in kas returning gRPC Unavailable. That affects our gRPC error ratio SLI and is not great.

Filtering by json.err = "HTTP Status code:" over the last 48 hours:

sum by (grpc_code,grpc_method,grpc_service) (
  increase(grpc_server_handled_total{app="kas",env="gprd",grpc_code!="OK",grpc_code!="Canceled",grpc_code!="NotFound",grpc_code!="FailedPrecondition",grpc_code!="Unauthenticated",grpc_code!="ResourceExhausted"}[5m])
)

added devopsconfigure [DEPRECATED] groupconfigure [DEPRECATED] sectionops labels

changed the description

Option 1

Retry requests that failed with a 5xx status code.

Agent is already retrying so this "masking" of errors is not great, on the other hand we are doing it in other places when polling Gitaly (albeit not for auth errors). I think this is the approach we should go for.

This was implemented, but in a different way - GitLab HTTP client now retries requests on failure using https://github.com/hashicorp/go-retryablehttp/.

Option 2

In cases where we know the error is caused by a failing upstream dependency, we can attach an extra bit of info into the returned error, inspect the error in the Prometheus metrics interceptor, and add a label to the error, indicating whether it's caused by an upstream or not.

I don't like this because it requires forking library code (Prometheus interceptors come from https://github.com/grpc-ecosystem/go-grpc-middleware) and tweaking it to our needs. It feels like a hack in general, included it for completeness.

changed the description

changed milestone to %14.3

assigned to @ash2k

mentioned in issue gitlab-com/gl-infra/production#5483 (closed)

marked this issue as related to gitlab-com/gl-infra/production#5483 (closed)

added typemaintenance label

added typefeature label

mentioned in merge request !496 (merged)

closed with merge request !496 (merged)

mentioned in commit 5b7ca0b9

marked this issue as related to gitlab-com/gl-infra/production#5520 (closed)

removed typefeature label

added groupenvironments label and removed groupconfigure [DEPRECATED] label