PARTIAL SUCCESS: Upgrade Geo multi-server installation from 12.10.12 to latest 13.0 version

Versions

Current: 12.10.12 Target: 13.0.10

Upgrade checklist

Preflight - upgrade

• Create a zoom meeting and schedule a recorded meeting
• Check if the Geo HA update instructions contain version specific changes
• Check if PostgreSQL is already the latest shipped version. If not, ensure PostgreSQL upgrade instructions are followed.
• Check if any upgrade warnings exist
• Verify that the Geo cluster is healthy pre-upgrade. For a no-downtime upgrade, remove deploy nodes from load balancers/stop sidekiq and run looping-pipeline to confirm test pass.
• Find the latest packaged version of GitLab that can be used for zero downtime upgrades
• Set up the looping test pipeline to run during the upgrade procedure.
• Open the HAProxy stats dashboard for each site, to monitor health checks

Upgrade

• Retrieve a beverage of choice within a drinkable temperature range
• Join Zoom meeting and wait for arrival. Hit the record button
• Manually trigger the looping test pipeline to start running smoke tests before upgrading the primary site and before upgrading the secondary site (if failures happen during the primary site update)
• Perform upgrade steps described in latest documentation.
• During the upgrade process, monitor HAProxy stats dashboard and the looping test pipeline for any failures
• Record any issues encountered during the upgrade
• Verify cluster health post upgrade
• Verify PostgreSQL version is correct

Postflight

• Record the upgrade outcome as SUCCESS (upgrade with zero downtime), FAILED, PARTIAL SUCCESS (upgrade but with downtime or unconfirmed downtime)
• Open new issues and inform @nhxnguyen and @fzimmer to confirm next steps
• Create new issue for the next upgrade demo (the next versions) and assign to @nhxnguyen and @fzimmer
• Update Geo validation tests docs page (doc/administration/geo/replication/geo_validation_tests.md)

added Quality backstage [DEPRECATED] devopssystems groupgeo labels

marked the checklist item Create a zoom meeting and schedule a recorded meeting as incomplete

marked the checklist item Check if PostgreSQL is already the latest shipped version. If not, ensure PostgreSQL upgrade instructions are followed. as incomplete

marked the checklist item Verify that the Geo cluster is healthy pre-upgrade. For a no-downtime upgrade, remove deploy nodes from load balancers/stop sidekiq and run looping-pipeline to confirm test pass. as incomplete

marked the checklist item Set up the looping test pipeline to run during the upgrade procedure. as incomplete

marked the checklist item Find the latest packaged version of GitLab that can be used for zero downtime upgrades as incomplete

marked the checklist item Retrieve a beverage of choice within a drinkable temperature range as incomplete

marked the checklist item Join Zoom meeting and wait for arrival. Hit the record button as incomplete

marked the checklist item Check if the Geo HA update instructions contain version specific changes as incomplete

marked the checklist item Check if any upgrade warnings exist as incomplete

marked the checklist item Perform upgrade steps described in latest documentation. as incomplete

marked the checklist item Manually trigger the looping test pipeline to start running smoke tests before upgrading the primary site and before upgrading the secondary site (if failures happen during the primary site update) as incomplete

marked this issue as related to #223284 (closed)

@jennielouie - Nothing to see here, I'm just doing a test for !34841 (comment 369799898)

@jennielouie - Actually you will have to reapply ~backstage if that's what you want since I won't be able to

removed backstage [DEPRECATED] label

added backstage [DEPRECATED] label

Hey @kwiebers, ~backstage is being deprecated in favor of featureaddition, ~"feature::maintenance", ~"tooling::pipelines", and ~"tooling::workflow" to improve the identification of these type of changes. Please see https://about.gitlab.com/handbook/engineering/management/throughput/#implementation for further guidance.

removed backstage [DEPRECATED] label

mentioned in merge request !34841 (merged)

added maintenanceworkflow label

added tooling (archive) label

changed the description

assigned to @nhxnguyen and @fzimmer

removed tooling (archive) label

added 1 deleted label and removed maintenanceworkflow label

changed the description

Tentatively setting to %13.3 for when we've finished with Gitaly HA and Patroni testing.

changed milestone to %13.3

unassigned @fzimmer and @nhxnguyen

added [deprecated] Accepting merge requests label

marked the checklist item Create a zoom meeting and schedule a recorded meeting as completed

marked the checklist item Check if the Geo HA update instructions contain version specific changes as completed

marked the checklist item Open the HAProxy stats dashboard for each site, to monitor health checks as completed

marked the checklist item Find the latest packaged version of GitLab that can be used for zero downtime upgrades as completed

marked the checklist item Set up the looping test pipeline to run during the upgrade procedure. as completed

marked the checklist item Check if any upgrade warnings exist as completed

marked the checklist item Check if PostgreSQL is already the latest shipped version. If not, ensure PostgreSQL upgrade instructions are followed. as completed

marked the checklist item Verify that the Geo cluster is healthy pre-upgrade. For a no-downtime upgrade, remove deploy nodes from load balancers/stop sidekiq and run looping-pipeline to confirm test pass. as completed

marked the checklist item Retrieve a beverage of choice within a drinkable temperature range as completed

marked the checklist item Join Zoom meeting and wait for arrival. Hit the record button as completed

marked the checklist item Manually trigger the looping test pipeline to start running smoke tests before upgrading the primary site and before upgrading the secondary site (if failures happen during the primary site update) as completed

marked the checklist item Perform upgrade steps described in latest documentation. as completed

marked the checklist item During the upgrade process, monitor HAProxy stats dashboard and the looping test pipeline for any failures as completed

marked the checklist item Record any issues encountered during the upgrade as completed

marked the checklist item Verify cluster health post upgrade as completed

marked the checklist item Verify PostgreSQL version is correct as completed

assigned to @jennielouie

added workflowin dev label

changed title from Upgrade Geo multi-server installation from 12.10.12 to latest 13.0 version to PARTIAL SUCCESS: Upgrade Geo multi-server installation from 12.10.12 to latest 13.0 version

marked the checklist item Record the upgrade outcome as SUCCESS (upgrade with zero downtime), FAILED, PARTIAL SUCCESS (upgrade but with downtime or unconfirmed downtime) as completed