Add information that Geo shouldn't be used for certain OS upgrades to docs
Problem to solve
We've had an increase of customers using Geo to upgrade from RHEL 7 (and friends) to RHEL 8, which is an unsupported upgrade method. The requirements for running Geo state that you need to check OS locale data compatibility before installing and configuring Geo; on RHEL 7 and RHEL 8, this check will return a result that shows they're incompatible.
Even still, we've seen at least a dozen customers running this configuration with the intent to failover to the RHEL 8 secondary, since RHEL 7 end-of-life is on 2024-06-30. This can cause long-term problems with GitLab, which may be hard to troubleshoot or remediate in the future.
Further details
As the documentation above details, this is the result of distributions upgrading the bundled version of glibc
from a pre-2.28 version to version 2.28 or newer. This changes the way sorting is handled, which PostgreSQL relies on for database indexes. Running Geo on two incompatible operating systems causes indexes on the secondary site to become silently corrupted, which causes issues with replication. These replication failures can be extremely difficult, confusing, and time consuming to attempt to troubleshoot.
Proposal
We should add documentation that states:
- You should not use Geo for an OS upgrade if the locale compatibility check fails.
- Other viable upgrade options.
- How to recover if you did complete an incompatible OS upgrade using Geo.
Who can address the issue
- The Geo team can help with specifics and writing the documentation
- The Support team can help gather examples of customers encountering this, so that we can ensure we cover as many use cases as possible
- The Technical Writing team can help with formatting and placing the content in the most impactful way
Other links/references
List of affected tickets: