Concurrent elections perform multiple failovers

Multiple Praefects may run the election query concurrently. This can occur with concurrent mutator RPCs that concurrently attempt to get the primary. The desired behavior is that only one of the queries would end up electing the primary and the second query would take the changes of the first query in to consideration, not elect a new primary and return the newly elected primary by the first query. This is not currently the case and both queries will independently elect a primary, causing the first elected primary to be dropped in favor of the second query.

This could cause primary flickering if the first primary manages to accept a write before the second one. Praefect would immediately failover back to it as the second primary would be outdated. Writes are only acknowledged when they end up on fully up to date nodes, so this should not lead to acknowledged writes being lost.

MR that fixes the test suite and causes failing tests: !4081 (916a5514)

Edited by Sami Hiltunen