Use repository generations to determine the best leader to elect
The current leader scoring logic uses the replication queue to determine the best new leader and considers every replication job equally. When electing a new leader, we should focus on minimizing data loss. When it comes to replication jobs, we should not consider jobs that do not indicate data loss such as repack and gc.
Using the replication queue as a data source also poses problems in combination with #2941 (closed), as we'll have to keep the dead jobs around to correctly account for unreplicated writes.
We could solve both of these problems by using the repository generation numbers for the scoring logic instead. Generation numbers should only increment when references are modified (pending #2977 (closed)), and as such, we'd automatically ignore repack and gc. From the generations, we can also determine how many writes the repository is behind without having to keep the replication job history around.
To do the scoring, we could do either of the following:
- Count the number of repositories that are behind on each storage. This would minimize the number of repositories that would go in to read-only mode following the failover.
- Count the number of generations the storage is behind in total. This would minimize the total data loss and would likely be the safer choice.
cc @zj-gitlab