~1.5 million gitaly Canceled gRPC requests across all shards and every gRPC method in the past week and the sudden drop-off of their occurence rate

There were roughly ~1.5 million gitaly gRPC requests which resulted in grpc.code: Canceled across all shards and 21 different gRPC methods over the past week.

Note that all logged occurrences were at the info level and not error.

Then, around 2020-12-27 12:00 UTC there was a sudden drop-off in the rate of these messages.

The most frequent gRPC method offender was PostUploadPack. This remains the case even after the rate drop-off.

Prometheus accounts for these gRPC requests with grpc.code: Canceled as errors, apparentlty:

The logged occurrence rate drop-off has not yet been explained.