Skip to content

Add gRPC exponential push back header

gRPC has a built-in mechanism for push-back: https://github.com/grpc/proposal/blob/master/A6-client-retries.md#pushback. This mechanism sets a special header grpc-retry-pushback-ms. It's a strong hint that the client must back off for a while before retrying again. Typically, clients with official gRPC libraries honor this header. One good thing of this mechanism is that, gRPC server has full power to determine this value. Clients supply grpc-previous-rpc-attempts header by default. Server can push back exponentially, or reject requests permanently.

In our use cases, Gitaly server can enable exponential retry on some certain RPCs. It can even config the initial back off, exponential factors, and retriable codes, etc. thanks to service config:

"retryPolicy": {
  "maxAttempts": 4,
  "initialBackoff": "1s",
  "maxBackoff": "5s",
  "backoffMultiplier": 2,
  "retryableStatusCodes": [
    "RESOURCE_EXHAUSETED"
  ]
}

More about this mechanism here: https://github.com/grpc/proposal/blob/master/A6-client-retries.md.

This is a typical flow when setting this header:

sequenceDiagram
  Users ->> InternalClients: Call
  InternalClients ->> GitalyServer : PostUploadPack
  GitalyServer ->> InternalClients: ResourceExhausted [grpc-retry-pushback-ms=1000]
  Note over InternalClients: Sleep
  InternalClients ->> GitalyServer : PostUploadPack
  GitalyServer ->> InternalClients: ResourceExhausted [grpc-retry-pushback-ms=1500]
  Note over InternalClients: Sleep
  InternalClients ->> GitalyServer : PostUploadPack
  GitalyServer ->> InternalClients: Data
  InternalClients ->> Users: Data

If the server rejects the request permanently:

sequenceDiagram
  Users ->> InternalClients: Call
  InternalClients ->> GitalyServer : PostUploadPack
  GitalyServer ->> InternalClients: ResourceExhausted [grpc-retry-pushback-ms=1000]
  Note over InternalClients: Sleep
  InternalClients ->> GitalyServer : PostUploadPack
  GitalyServer ->> InternalClients: ResourceExhausted [grpc-retry-pushback-ms=1700]
  Note over InternalClients: Sleep
  InternalClients ->> GitalyServer : PostUploadPack
  GitalyServer ->> InternalClients: ResourceExhausted [grpc-retry-pushback-ms=3500]
  Note over InternalClients: Sleep
  InternalClients ->> GitalyServer : PostUploadPack
  GitalyServer ->> InternalClients: ResourceExhausted
  InternalClients ->> Users: 429 status
Edited by Quang-Minh Nguyen
To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information