Geo: Simplify Geo upgrade process
<!-- triage-serverless v3 PLEASE DO NOT REMOVE THIS SECTION -->
*This page may contain information related to upcoming products, features and functionality.
It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes.
Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc.*
<!-- triage-serverless v3 PLEASE DO NOT REMOVE THIS SECTION -->
Engineering Owner: @toon
### Summary (for the Release Post; describes potential future state)
Geo upgrades required a large number of manual steps when upgrading GitLab in regular and high-availability configurations. For systems administrators, this was cumbersome and could cause issues when upgrading Geo. Geo upgrades now require fewer manual steps when upgrading a high-availability installation (from X to X). Geo upgrades in regular installations now do not require any additional manual intervention. We have improved the documentation to guide users through different deployment modes (regular, high-availability, high-availability + zero downtime upgrades)
### Problem to solve
Upgrading Geo is complicated and highly-manual. This may result in issues when upgrading because of user errors and can generally decrease the willingness of GitLab customers to upgrade to more recent installations. There should be as little effort as possible for customers to upgrade to the latest version of GitLab and Geo.
Overall, I would suggest that the desired end state is that the GitLab update mechanism detects if GitLab is run using Geo in **primary** or **secondary** mode and what kind of deployment is configured (HA/ zero downtime) and then performs all additionally steps automatically, if at all possible. This would mean, that Geo upgrades require no or very little manual intervention by system administrators.
If this is not possible, the documentation needs to clearly reflect all necessary steps.
#### Geo upgrade in a regular configuration
Currently, updating Geo requires updating GitLab itself on every **primary** and **secondary** nodes, followed by two manual steps:
1. [Updating the tracking database on secondary node(s)](https://docs.gitlab.com/ee/administration/geo/replication/updating_the_geo_nodes.html#update-tracking-database-on-secondary-node)
1. [Testing **primary** and **secondary** nodes + version checks](https://docs.gitlab.com/ee/administration/geo/replication/updating_the_geo_nodes.html#check-status-after-updating)
#### Geo upgrade in a high-availability configuration
[Upgrading Geo in a HA configuration](https://docs.gitlab.com/omnibus/update/#geo-deployment) requires eight manual steps for the *primary*() and eight manual steps *per secondary*. For large Geo deployments, this means a systems administrator will need to execute the same set of instructions several times, which is inconvenient.
For example, a large customer with eight secondaries and one primary will need to **execute 72 steps** in total!
### Proposal
- Reduce the number of manual steps in non-HA and HA solutions
- Where possible, automate the Geo upgrade
- Clean up the documentation for Geo upgrades and clearly describe steps for
- Regular Geo deployment
- HA deployment
- HA deployment with zero downtime upgrades
### Higher intent
In line with our current strategy fo Geo, we are focusing on making Geo easier to install, maintain and upgrade. Geo users rely on Geo functionalities for either or both Disaster Recovery and Geo Replication. These functionalities are business critical and we need to ensure a high-level of customer trust. Creating a system that is robust and easy to operate is an important step to improve the user experience.
### Intended users
* [Sydney, Systems Administrator](https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/#sidney-systems-administrator)
### Further details
<!-- Include use cases, benefits, and/or goals (contributes to our vision?) -->
As a systems administrator, I would like to update GitLab on my *primary* and all my *secondary* instances in the same way, so that I don't have to worry about different configurations.
As a systems administrator, I would like to be informed automatically if a problem with Geo occurs during the upgrade, so that I don't have to remember to check versions manually.
As a systems administrator, I would like database migrations to be run automatically when needed, so I don't have to remember to check manually.
### Permissions and Security
<!-- What permissions are required to perform the described actions? Are they consistent with the existing permissions as documented for users, groups, and projects as appropriate? Is the proposed behavior consistent between the UI, API, and other access methods (e.g. email replies)? -->
N/A
### Documentation
<!-- See the Feature Change Documentation Workflow https://docs.gitlab.com/ee/development/documentation/feature-change-workflow.html
Add all known Documentation Requirements here, per https://docs.gitlab.com/ee/development/documentation/feature-change-workflow.html#documentation-requirements -->
Updates are required to: https://docs.gitlab.com/ee/administration/geo/replication/updating_the_geo_nodes.html and https://docs.gitlab.com/omnibus/update/#geo-deployment
Overall, the documentation needs to be improved for HA deployments and HA + Zero downtime employments.
### Testing
<!-- What risks does this change pose? How might it affect the quality of the product? What additional test coverage or changes to tests will be needed? Will it require cross-browser testing? See the test engineering process for further guidelines: https://about.gitlab.com/handbook/engineering/quality/guidelines/test-engineering/ -->
Increased automation often requires more testing in order to establish that operations succeeded.
### What does success look like, and how can we measure that?
<!-- Define both the success metrics and acceptance criteria. Note that success metrics indicate the desired business outcomes, while acceptance criteria indicate when the solution is working correctly. If there is no way to measure success, link to an issue that will implement a way to measure this. -->
Success means that the number of additional steps to update GitLab including Geo (both in *primary* and *secondary* installation is significantly reduced. Currently, two steps are needed. Success would be 1 or 0 steps.
### What is the type of buyer?
- Premium
### Links / references
epic