Skip to content

Consider primary modified only if a subtransaction was committed

Sami Hiltunen requested to merge smh-check-if-primary-was-modified into master

Praefect currently considers secondaries to be outdated after every mutator RPC in which they did not successfully commit the transaction. This causes Praefect to consider secondaries outdated even after RPCs that could have clearly not modified the primary, for example when the network connection was down and the request was not even sent to the network. This can cause secondaries to be considered outdated especially in failover scenarios, where there's a brief period when the primary's connection may be down before Praefect's health check realizes it.

This MR addresses the situation by only considering the secondaries outdated if there could have been modifications to the primary replica. This is done by checking whether at least one subtranscation was committed during the transaction. As Praefect requires primary to always agree on the vote results, any committed subtransaction indicates the primary has likely performed some changes. If there are no committed subtransactions, then the primary would not have persisted reference updates to the disk.

This still doesn't guarantee the primary actually performed the changes, as Praefect simply instructs the Gitalys to commit without waiting for a response. For example, the primary could receive the commit message and immediately crash. In such cases the primary would perform no changes and the secondaries would still be considered outdated. Using the subtransactions to determine whether changes could have been made does reduce the occurrence of this problem in a major way though.

As a special case, if a transactional RPC succeeds without any subtransactions, we'll still replicate on every secondary. Every transactional RPC should have voted on success, so this indicates something is very off.

For non-transactional RPCs, we still don't know for sure if the primary performed changes or not. As such, we still need to consider the secondaries outdated after failed RPCs. Most of these RPCs do not result an update type replication job and would not increment the generation anyway. Some do, which we may want to address in follow-ups.

Edited by Sami Hiltunen

Merge request reports