Skip to content

Extend retry-period for read-only Gitaly RPCs

Will Chandler requested to merge wc/extend-gitaly-retries into master

What does this MR do and why?

Extend retry-period for read-only Gitaly RPCs

With !133896 (merged) we configured the Gitaly client to send up to two retries on read-only RPCs, waiting no more than 750ms in total.

After rolling out the new Gitaly upgrade process to canary, we have observed that Gitaly restarts may take up to two seconds to begin accepting traffic, more than the grace period configured for retries.

Expand the retry policy to allow up to three retries with a starting delay of 0.33 seconds and a maximum backoff of 1.5 seconds. This gives us a total grace period of 2.3 seconds, which is still well within the default deadline of 10 seconds.

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

How to set up and validate locally

  1. Stop your GDK's Gitaly with gdk stop gitaly
  2. Load a rails console session and execute a FindCommit with Project.last.repository.commit('HEAD')
  3. Immediately start Gitaly with gdk start gitaly. If Gitaly starts fast enough, the request will succeed with output like => #<Commit id:603975295665c2601289682bd3eefe92da22f848 i-user-0-1696879720/lab-coat@603975295665c2601289682bd3eefe92da22f848>
    1. If you have trouble getting the timing right, increasing maxAttempts and maxBackoff may help.
Edited by Will Chandler

Merge request reports