Add gRPC exponential push back header
gRPC has a built-in mechanism for push-back: https://github.com/grpc/proposal/blob/master/A6-client-retries.md#pushback. This mechanism sets a special header grpc-retry-pushback-ms
. It's a strong hint that the client must back off for a while before retrying again. Typically, clients with official gRPC libraries honor this header. One good thing of this mechanism is that, gRPC server has full power to determine this value. Clients supply grpc-previous-rpc-attempts
header by default. Server can push back exponentially, or reject requests permanently.
In our use cases, Gitaly server can enable exponential retry on some certain RPCs. It can even config the initial back off, exponential factors, and retriable codes, etc. thanks to service config:
"retryPolicy": {
"maxAttempts": 4,
"initialBackoff": "1s",
"maxBackoff": "5s",
"backoffMultiplier": 2,
"retryableStatusCodes": [
"RESOURCE_EXHAUSETED"
]
}
More about this mechanism here: https://github.com/grpc/proposal/blob/master/A6-client-retries.md.
This is a typical flow when setting this header:
sequenceDiagram
Users ->> InternalClients: Call
InternalClients ->> GitalyServer : PostUploadPack
GitalyServer ->> InternalClients: ResourceExhausted [grpc-retry-pushback-ms=1000]
Note over InternalClients: Sleep
InternalClients ->> GitalyServer : PostUploadPack
GitalyServer ->> InternalClients: ResourceExhausted [grpc-retry-pushback-ms=1500]
Note over InternalClients: Sleep
InternalClients ->> GitalyServer : PostUploadPack
GitalyServer ->> InternalClients: Data
InternalClients ->> Users: Data
If the server rejects the request permanently:
sequenceDiagram
Users ->> InternalClients: Call
InternalClients ->> GitalyServer : PostUploadPack
GitalyServer ->> InternalClients: ResourceExhausted [grpc-retry-pushback-ms=1000]
Note over InternalClients: Sleep
InternalClients ->> GitalyServer : PostUploadPack
GitalyServer ->> InternalClients: ResourceExhausted [grpc-retry-pushback-ms=1700]
Note over InternalClients: Sleep
InternalClients ->> GitalyServer : PostUploadPack
GitalyServer ->> InternalClients: ResourceExhausted [grpc-retry-pushback-ms=3500]
Note over InternalClients: Sleep
InternalClients ->> GitalyServer : PostUploadPack
GitalyServer ->> InternalClients: ResourceExhausted
InternalClients ->> Users: 429 status