Skip to content

gitaly_client: Fix indeterministic cherry-picks and reverts

Patrick Steinhardt requested to merge pks-gitaly-deterministic-cherry-pick-and-revert into master

What does this MR do and why?

When calling RPCs in Gitaly Cluster, all nodes backing the cluster will typically perform the same change and then vote on the result -- and if the election gains quorum, the commit is committed to disk. It is thus mandatory that all nodes arrive at the same result without any kind of indeterminism.

One source of indeterminism is the current time though. By default, RPCs like UserMergeBranch, UserCherryPick or UserRevert will use the current time as committer date for the newly created commit. While fine in a standalone Gitaly setup where there is only a single source of time, it is not fine in Gitaly Cluster where the request gets served by multiple nodes at once. If their time diverges or if the nodes aren't going in the exact same pace, then it is quite likely that they will end up with different commits because Time.current returns different results. The consequence is that the transaction will not reach quorum and thus the change will not be performed.

To fix this, Gitaly has introduced a timestamp field for all of its RPCs that use the system time. This field needs to be set by clients, and, if set, will override the current system time in favor of the time sent by the client. This means that the source of indeterminism is fixed from Gitaly's perspective as all nodes are guaranteed to use the same committer date now, and thus they should end up with the same commit.

While we have migrated Rails to set this field for most of the RPCs, we didn't update UserRevert and UserCherryPick to set it yet, and this is causing flaky QA tests. So let's fix this oversight and also set up this field such that things become deterministic.

Fixes Failure in browser_ui/3_create/merge_request/ch... (#421712 - closed).

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Merge request reports