Document lazy failovers in Praefect

What does this MR do?

• Praefect has stopped eagerly performing failovers and now only elects a new primary if the current one is unhealthy and a primary is needed to serve a request. This commit documents the new failover behavior.

• Praefect's failover strategy has become more flexible with recent changes. These changes have done away with the read-only mode. The read-only mode blocked writes to a repository if the primary was outdated. With the new failover logic, Praefect will instead failover from the outdated primary to an up to date one. Given this, the read-only mode no longer exists. This commit removes references to the read-only mode from Praefect's documentation.

• Praefect's dataloss now includes in the output repositories which are unavailable on the cluster. With variable replication factor, it's no longer sufficient to only list the repositories which have outdated, assigned replicas. A repository with replication factor of one may be fully up to date but may be unavailable if the only replica is unhealthy. This amounts to dataloss if the Gitaly node is not brought back online.

  Praefects lazy failover behavior has also made the read-only mode non-existent, as Praefect now immediately fails over to an up to date replica if one is available. Praefect also no longer elects outdated replicas as primaries. Given these changes, this commit updates the dataloss documentation to reflect these behaviors.

Related issues

• gitaly!3542 (merged)
• gitaly!3543 (merged)
• gitaly!3548 (merged)
• gitaly!3548 (merged)

Author's checklist (required)

• Follow the Documentation Guidelines and Style Guide.
• If you have Developer permissions or higher:
  • Ensure that the product tier badge is added to doc's h1.
  • Apply the documentation label, plus:
    • The corresponding DevOps stage and group labels, if applicable.
    • development guidelines when changing docs under doc/development/*, CONTRIBUTING.md, or README.md.
    • development guidelines and Documentation guidelines when changing docs under development/documentation/*.
    • development guidelines and Description templates (.gitlab/*) when creating/updating issue and MR description templates.
  • Request a review from the designated Technical Writer.

Do not add the feature, frontend, backend, ~"bug", or database labels if you are only updating documentation. These labels will cause the MR to be added to code verification QA issues.

When applicable:

• Update the permissions table.
• Link docs to and from the higher-level index page, plus other related docs where helpful.
• Add the product tier badge accordingly.
• Add GitLab's version history note(s).
• Add/update the feature flag section.

Review checklist

All reviewers can help ensure accuracy, clarity, completeness, and adherence to the Documentation Guidelines and Style Guide.

1. Primary Reviewer

• Review by a code reviewer or other selected colleague to confirm accuracy, clarity, and completeness. This can be skipped for minor fixes without substantive content changes.

2. Technical Writer

• Technical writer review. If not requested for this MR, must be scheduled post-merge. To request for this MR, assign the writer listed for the applicable DevOps stage.
  • Ensure docs metadata are present and up-to-date.
  • Ensure Technical Writing and documentation are added.
  • Add the corresponding docs:: scoped label.
  • If working on UI text, add the corresponding UI Text scoped label.
  • Add twdoing when starting work on the MR.
  • Add twfinished if Technical Writing team work on the MR is complete but it remains open.

For more information about labels, see Technical Writing workflows - Labels.

For suggestions that you are confident don't need to be reviewed, change them locally and push a commit directly to save others from unneeded reviews. For example:

• Clear typos, like this is a typpo.
• Minor issues, like single quotes instead of double quotes, Oxford commas, and periods.

For more information, see our documentation on Merging a merge request.

3. Maintainer

1. Review by assigned maintainer, who can always request/require the above reviews. Maintainer's review can occur before or after a technical writer review.
2. Ensure a release milestone is set.
3. If there has not been a technical writer review, create an issue for one using the Doc Review template.