GSTG upgrade of Patroni from 3.3.0 → 3.3.4 on the gstg staging registry cluster

Production Change

Change Summary

Rolling zero-downtime upgrade of Patroni from 3.3.0 → 3.3.4 on the gstg-patroni-registry-v17 cluster in gstg.

Each node is taken out of load balance individually via nofailover + noloadbalance tags, Patroni is upgraded and restarted, the node is returned to the pool, and a controlled switchover handles the primary before it is upgraded last.

  • Current version: 3.3.0
  • Target version: 3.3.4

Change Details

Cluster Layout

Member Host Role State
patroni-registry-v17-01-db-gstg 10.224.117.101 Leader running
patroni-registry-v17-02-db-gstg 10.224.117.102 Replica streaming
patroni-registry-v17-03-db-gstg 10.224.117.103 Replica streaming
patroni-registry-v17-04-db-gstg 10.224.117.104 Replica streaming

Note: Node 02 already has nofailover/noloadbalance tags. Node 01 is the current primary and will be upgraded last.

Important

If your change involves scheduled maintenance, add a step to set and unset maintenance mode per our runbooks. This will make sure SLA calculations adjust for the maintenance period.

Preparation

Note

The following checklists must be done in advance, before setting the label changescheduled

Change Reviewer checklist

C4 C3 C2 C1:

  • Check if the following applies:
    • The scheduled day and time of execution of the change is appropriate.
    • The change plan is technically accurate.
    • The change plan includes estimated timing values based on previous testing.
    • The change plan includes a viable rollback plan.
    • The specified metrics/monitoring dashboards provide sufficient visibility for the change.

C2 C1:

  • Check if the following applies:
    • The complexity of the plan is appropriate for the corresponding risk of the change. (i.e. the plan contains clear details).
    • The change plan includes success measures for all steps/milestones during the execution.
    • The change adequately minimizes risk within the environment/service.
    • The performance implications of executing the change are well-understood and documented.
    • The specified metrics/monitoring dashboards provide sufficient visibility for the change.
      • If not, is it possible (or necessary) to make changes to observability platforms for added visibility?
    • The change has a primary and secondary SRE with knowledge of the details available during the change window.
    • The change window has been agreed with Release Managers in advance of the change. If the change is planned for APAC hours, this issue has an agreed pre-change approval.
    • The labels blocks deployments and/or blocks feature-flags are applied as necessary.

Change Technician checklist

  • The Change Criticality has been set appropriately and requirements have been reviewed.
  • The change plan is technically accurate.
  • The rollback plan is technically accurate and detailed enough to be executed by anyone with access.
  • This Change Issue is linked to the appropriate Issue and/or Epic
  • Change has been tested in staging and results noted in a comment on this issue.
  • A dry-run has been conducted and results noted in a comment on this issue.
  • The change execution window respects the Production Change Lock periods.
  • Once all boxes above are checked, mark the change request as scheduled: /label ~"change::scheduled"
  • For C1 and C2 change issues, the change event is added to the GitLab Production calendar by the change-scheduler bot. It is schedule to run every 2 hours.
  • For C1 and C2 change issues, Platform Leadership provides approval with the platform_leadership_approved label on the issue. Mention @gitlab-org/saas-platforms/change-review-leadership in this issue with a reference to review guidelines to get approval and provide visibility to all infrastructure managers.
  • For C1, C2, or blocks deployments change issues, confirm with Release managers that the change does not overlap or hinder any release process (In #production channel, mention @release-managers and this issue and await their acknowledgment.)
  • For C1 change issues or C2 change issues happening during weekend, SREs on-call must be informed at least 2 weeks in advance. Check the incident.io GitLab.com Production EOC schedule to find who will be on-call at the scheduled day and time.

Detailed steps for the change

Pre-execution steps

Note

The following steps should be done right at the scheduled time of the change request. The preparation steps are listed below.

  • Make sure all tasks in Change Technician checklist are done
  • For C1 and C2 change issues, the SRE on-call has been informed prior to change being rolled out.
    • The SRE on-call provided approval with the eoc_approved label on the issue.
  • For C1, C2, or blocks deployments change issues, Release managers have been informed prior to change being rolled out. (In #production channel, mention @release-managers and this issue and await their acknowledgment.)
  • There are currently no active incidents that are severity1 or severity2
  • If the change involves doing maintenance on a database host, an appropriate silence targeting the host(s) should be added for the duration of the change.

Change steps - steps to take to execute the change

Estimated Time to Complete (mins) - 240

Pre-flight Checks

gitlab-patronictl list
  • Confirm current Patroni version on all nodes
knife ssh "roles:gstg-base-db-patroni-registry-v17" "sudo gitlab-patronictl version"
# Expected: patronictl version 3.3.0

Execution Steps

  • Set label change in-progress /label changein-progress
  • Pause cluster maintenance mode
gitlab-patronictl pause gstg-patroni-registry-v17
gitlab-patronictl list
# Confirm: Maintenance mode: on

Replica: patroni-registry-v17-02-db-gstg

  • Create Alertmanager silence for the node
ssh console-01-sv-gstg.c.gitlab-staging-1.internal
python3 /usr/local/bin/alert_management.py create \
  --environment gstg \
  --fqdn "patroni-registry-v17-02-db-gstg.*" \
  --duration-hours 1 \
  --comment "Patroni upgrade 3.3.0 -> 3.3.4 - https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21885"
  • Add maintenance role to node
knife node run_list add patroni-registry-v17-02-db-gstg.c.gitlab-staging-1.internal "role[gstg-base-db-patroni-maintenance]"
  • Run chef-client
knife ssh "name:patroni-registry-v17-02-db-gstg.c.gitlab-staging-1.internal" "sudo chef-client"
ssh patroni-registry-v17-02-db-gstg.c.gitlab-staging-1.internal
sudo systemctl restart patroni
  • Verify new version, clean restart, and node is excluded from LB
gitlab-patronictl version
# Expected: patronictl version 3.3.4
journalctl -u patroni -n 20 --no-pager
gitlab-patronictl list
# Node should show nofailover/noloadbalance tags
  • Remove maintenance role from node
knife node run_list remove patroni-registry-v17-02-db-gstg.c.gitlab-staging-1.internal "role[gstg-base-db-patroni-maintenance]"
  • Run chef-client to clear maintenance tags
knife ssh "name:patroni-registry-v17-02-db-gstg.c.gitlab-staging-1.internal" "sudo chef-client"
  • Expire Alertmanager silence for the node
ssh console-01-sv-gstg.c.gitlab-staging-1.internal
python3 /usr/local/bin/alert_management.py expire --silence-id <silence-id>
  • Confirm replication lag is 0 and node is healthy before continuing
gitlab-patronictl list

Replica: patroni-registry-v17-03-db-gstg

  • Create Alertmanager silence for the node
ssh console-01-sv-gstg.c.gitlab-staging-1.internal
python3 /usr/local/bin/alert_management.py create \
  --environment gstg \
  --fqdn "patroni-registry-v17-03-db-gstg.*" \
  --duration-hours 1 \
  --comment "Patroni upgrade 3.3.0 -> 3.3.4 - https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21885"
  • Add maintenance role to node
knife node run_list add patroni-registry-v17-03-db-gstg.c.gitlab-staging-1.internal "role[gstg-base-db-patroni-maintenance]"
  • Run chef-client
knife ssh "name:patroni-registry-v17-03-db-gstg.c.gitlab-staging-1.internal" "sudo chef-client"
ssh patroni-registry-v17-03-db-gstg.c.gitlab-staging-1.internal
sudo systemctl restart patroni
  • Verify new version, clean restart, and node is excluded from LB
gitlab-patronictl version
journalctl -u patroni -n 20 --no-pager
gitlab-patronictl list
  • Remove maintenance role from node
knife node run_list remove patroni-registry-v17-03-db-gstg.c.gitlab-staging-1.internal "role[gstg-base-db-patroni-maintenance]"
  • Run chef-client to clear maintenance tags
knife ssh "name:patroni-registry-v17-03-db-gstg.c.gitlab-staging-1.internal" "sudo chef-client"
  • Expire Alertmanager silence for the node
ssh console-01-sv-gstg.c.gitlab-staging-1.internal
python3 /usr/local/bin/alert_management.py expire --silence-id <silence-id>
  • Confirm replication lag is 0 and node is healthy before continuing
gitlab-patronictl list

Replica: patroni-registry-v17-04-db-gstg

  • Create Alertmanager silence for the node
ssh console-01-sv-gstg.c.gitlab-staging-1.internal
python3 /usr/local/bin/alert_management.py create \
  --environment gstg \
  --fqdn "patroni-registry-v17-04-db-gstg.*" \
  --duration-hours 1 \
  --comment "Patroni upgrade 3.3.0 -> 3.3.4 - https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21885"
  • Add maintenance role to node
knife node run_list add patroni-registry-v17-04-db-gstg.c.gitlab-staging-1.internal "role[gstg-base-db-patroni-maintenance]"
  • Run chef-client
knife ssh "name:patroni-registry-v17-04-db-gstg.c.gitlab-staging-1.internal" "sudo chef-client"
ssh patroni-registry-v17-04-db-gstg.c.gitlab-staging-1.internal
sudo systemctl restart patroni
  • Verify new version, clean restart, and node is excluded from LB
gitlab-patronictl version
journalctl -u patroni -n 20 --no-pager
gitlab-patronictl list
  • Remove maintenance role from node
knife node run_list remove patroni-registry-v17-04-db-gstg.c.gitlab-staging-1.internal "role[gstg-base-db-patroni-maintenance]"
  • Run chef-client to clear maintenance tags
knife ssh "name:patroni-registry-v17-04-db-gstg.c.gitlab-staging-1.internal" "sudo chef-client"
  • Expire Alertmanager silence for the node
ssh console-01-sv-gstg.c.gitlab-staging-1.internal
python3 /usr/local/bin/alert_management.py expire --silence-id <silence-id>
  • Confirm replication lag is 0 and node is healthy before continuing
gitlab-patronictl list

Primary Switchover

Confirm patroni-registry-v17-01-db-gstg is still the primary before proceeding.

  • Confirm current primary
gitlab-patronictl list
  • Disable cluster maintenance mode
gitlab-patronictl resume gstg-patroni-registry-v17
gitlab-patronictl list
  • Run switchover using the switchover playbook
ssh console-01-sv-gstg.c.gitlab-staging-1.internal
sudo su - dbupgrade
test -d ~/src/db-migration || git clone https://gitlab.com/gitlab-com/gl-infra/db-migration.git ~/src/db-migration
cd src/db-migration
git checkout master
git pull
python3 -m venv ansible
source ansible/bin/activate
python3 -m pip install --upgrade pip
python3 -m pip install ansible
ansible --version
export PYTHONUNBUFFERED=1
cd dbre-toolkit
ansible-playbook -i inventory/gstg-registry.yml switchover_patroni_leader.yml 2>&1 | ts | tee -a ansible_switchover_patroni_leader_registry_$(date +%Y%m%d).log
  • Verify new primary is healthy and all replicas are replicating
gitlab-patronictl list
  • Re-pause cluster maintenance mode
gitlab-patronictl pause gstg-patroni-registry-v17
gitlab-patronictl list
# Confirm: Maintenance mode: on

Old Primary: patroni-registry-v17-01-db-gstg (now replica)

  • Create Alertmanager silence for the node
ssh console-01-sv-gstg.c.gitlab-staging-1.internal
python3 /usr/local/bin/alert_management.py create \
  --environment gstg \
  --fqdn "patroni-registry-v17-01-db-gstg.*" \
  --duration-hours 1 \
  --comment "Patroni upgrade 3.3.0 -> 3.3.4 - https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21885"
  • Add maintenance role to node
knife node run_list add patroni-registry-v17-01-db-gstg.c.gitlab-staging-1.internal "role[gstg-base-db-patroni-maintenance]"
  • Run chef-client
knife ssh "name:patroni-registry-v17-01-db-gstg.c.gitlab-staging-1.internal" "sudo chef-client"
ssh patroni-registry-v17-01-db-gstg.c.gitlab-staging-1.internal
sudo systemctl restart patroni
  • Verify new version and clean restart
gitlab-patronictl version
journalctl -u patroni -n 20 --no-pager
gitlab-patronictl list
  • Remove maintenance role from node
knife node run_list remove patroni-registry-v17-01-db-gstg.c.gitlab-staging-1.internal "role[gstg-base-db-patroni-maintenance]"
  • Run chef-client to clear maintenance tags
knife ssh "name:patroni-registry-v17-01-db-gstg.c.gitlab-staging-1.internal" "sudo chef-client"
  • Expire Alertmanager silence for the node
ssh console-01-sv-gstg.c.gitlab-staging-1.internal
python3 /usr/local/bin/alert_management.py expire --silence-id <silence-id>
  • Confirm all 4 nodes healthy, no replication lag
gitlab-patronictl list

Post-Upgrade Validation

  • All 4 nodes running 3.3.4
knife ssh "roles:gstg-base-db-patroni-registry-v17" "gitlab-patronictl version"
  • Disable cluster maintenance mode
gitlab-patronictl resume gstg-patroni-registry-v17
gitlab-patronictl list
  • gitlab-patronictl list — all members healthy, no replication lag
  • No unexpected failovers during the upgrade window
  • Monitor cluster for 1 hour
  • Set label change complete /label changecomplete

Rollback

If a node fails to restart after chef-client, revert the Chef MR and re-run chef-client on the affected node:

knife ssh "name:{AFFECTED_NODE}.c.gitlab-staging-1.internal" "sudo chef-client"
ssh {AFFECTED_NODE}.c.gitlab-staging-1.internal
sudo systemctl restart patroni

Ensure the maintenance role is removed from the affected node regardless:

knife node run_list remove {AFFECTED_NODE}.c.gitlab-staging-1.internal "role[gstg-base-db-patroni-maintenance]"

If the cluster enters a bad state, pause immediately:

gitlab-patronictl pause gstg-patroni-registry-v17

Monitoring

Key metrics to observe: Patroni Overview — https://dashboards.gitlab.net/d/patroni-registry-main/patroni-registry3a-overview?orgId=1&from=now-6h%2Fm&to=now%2Fm&timezone=utc&var-PROMETHEUS_DS=mimir-gitlab-gstg&var-environment=gstg

Edited by Saad Ullah