[GSTG] - Upgrade V14 nodes on clusters patroni-main-v14 and patroni-ci-v14 to N2

Production Change

Change Summary

This change objective is to rollout the hardware upgrade of our Patroni Main and CI clusters in GSTG from n1-standard-8 VM type to n2-standard-8 VM type.

The rollout plan was defined at https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/18934

As discussed in our last DBRE weekly meeting and at https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/18934#note_1358281302 we'll just upgrade the hardware of nodes on clusters patroni-main-v14 and patroni-ci-v14, which will then become the effective staging environment after the PG14 Upgrade on staging on 2023-04-26.​​

Change Details

  1. Services Impacted - ServicePatroni ServicePatroniCI
  2. Change Technician - @rhenchen.gitlab
  3. Change Reviewer - @alexander-sosna or @bshah11
  4. Time tracking - 6 hours
  5. Downtime Component - no downtime

Detailed steps for the change

Change Steps - steps to take to execute the change

Estimated Time to Complete (mins) - 4 hours (split in 2 days)

  1. Set label changein-progress /label ~change::in-progress
  2. Launch 4 new N2 nodes on each cluster - MR: https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/merge_requests/5543
  3. Start patroni on the newly added nodes 
    ssh_cluster_regex.sh "patroni-(ci|main)-v14-1..-db-gstg.*" "sudo systemctl start patroni"
  4. Replace Backup Nodes - MR: https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/merge_requests/5545
    • Execute the following in a chef-repo: 
      knife node run_list remove patroni-main-v14-06-db-gstg.c.gitlab-staging-1.internal "role[gstg-base-db-patroni-main-v14-backup-replica]"  
knife node run_list add patroni-main-v14-102-db-gstg.c.gitlab-staging-1.internal "role[gstg-base-db-patroni-main-v14-backup-replica]"
knife node run_list remove patroni-ci-v14-03-db-gstg.c.gitlab-staging-1.internal "role[gstg-base-db-patroni-ci-v14-backup-replica]"
knife node run_list add patroni-ci-v14-102-db-gstg.c.gitlab-staging-1.internal "role[gstg-base-db-patroni-ci-v14-backup-replica]"
knife search node "roles:gstg-base-db-patroni-main-v14" -a run_list
knife search node "roles:gstg-base-db-patroni-ci-v14" -a run_list
    • Force chef to run on the new backup nodes 
      ssh patroni-main-v14-102-db-gstg.c.gitlab-staging-1.internal "sudo chef-client"
ssh patroni-ci-v14-102-db-gstg.c.gitlab-staging-1.internal "sudo chef-client"
    • Execute the following to remove the script and cron from the old backup nodes patroni-main-v14-06-db-gstg.c.gitlab-staging-1.internal and patroni-ci-v14-03-db-gstg.c.gitlab-staging-1.internal 
      sudo rm /etc/gitlab/gcs-snapshot.json
sudo rm /usr/local/bin/gcs-snapshot.sh
sudo crontab -e -u gitlab-psql
      • Remove the following gitlab-psql user crontab lines 
        # Chef Name: GCS snapshot
PATH="/usr/local/sbin:/usr/sbin/:/sbin:/usr/local/bin:/usr/bin:/bin:/snap/bin"
0 * * * * /usr/local/bin/gcs-snapshot.sh
  5. Test Backup on the new backup nodes
    1. Log into patroni-main-v14-102 and patroni-ci-v14-102
    2. Execute:
    sudo su - gitlab-psql
PATH="/usr/local/sbin:/usr/sbin/:/sbin:/usr/local/bin:/usr/bin:/bin:/snap/bin"
/usr/local/bin/gcs-snapshot.sh
  6. Switchover the Primary Standby Leader node from node 01 to node 101 on each cluster
    • Execute the following command on any cluster node, and make sure to select patroni-main-v14-101 and patroni-ci-v14-101 as candidates for the switchover; 
      sudo gitlab-patronictl switchover
    • Then check the cluster status 
      sudo gitlab-patronictl list
  7. Mark N1 node as to_be_destroyed - MR: https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/merge_requests/5547
    • Execute for Main cluster 
      
knife node run_list add patroni-main-v14-01-db-gstg.c.gitlab-staging-1.internal "role[gstg-base-db-patroni-to_be_destroyed]"
knife node run_list add patroni-main-v14-02-db-gstg.c.gitlab-staging-1.internal "role[gstg-base-db-patroni-to_be_destroyed]"
knife node run_list add patroni-main-v14-03-db-gstg.c.gitlab-staging-1.internal "role[gstg-base-db-patroni-to_be_destroyed]"
knife node run_list add patroni-main-v14-06-db-gstg.c.gitlab-staging-1.internal "role[gstg-base-db-patroni-to_be_destroyed]"
knife node run_list add patroni-main-v14-07-db-gstg.c.gitlab-staging-1.internal "role[gstg-base-db-patroni-to_be_destroyed]"
knife search node "roles:gstg-base-db-patroni-main-v14" -a run_list
knife ssh "roles:gstg-base-db-patroni-main-v14" "sudo chef-client"
    • Execute for CI cluster 
      knife node run_list add patroni-ci-v14-01-db-gstg.c.gitlab-staging-1.internal "role[gstg-base-db-patroni-to_be_destroyed]"
knife node run_list add patroni-ci-v14-02-db-gstg.c.gitlab-staging-1.internal "role[gstg-base-db-patroni-to_be_destroyed]"
knife node run_list add patroni-ci-v14-03-db-gstg.c.gitlab-staging-1.internal "role[gstg-base-db-patroni-to_be_destroyed]"
knife node run_list add patroni-ci-v14-04-db-gstg.c.gitlab-staging-1.internal "role[gstg-base-db-patroni-to_be_destroyed]"
knife node run_list add patroni-ci-v14-05-db-gstg.c.gitlab-staging-1.internal "role[gstg-base-db-patroni-to_be_destroyed]"
knife search node "roles:gstg-base-db-patroni-ci-v14" -a run_list
knife ssh "roles:gstg-base-db-patroni-ci-v14" "sudo chef-client"
  8. Set label changescheduled /label ~change::scheduled
  9. Grace period of 1 day
  10. Set label changein-progress /label ~change::in-progress
  11. Destroy the N1 nodes marked to_be_destroyed and set new default machine_type: patroni-n2-standard-8 for TF patroni-main-v14 and patroni-ci-v14 modules - MR: https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/merge_requests/5548
  12. Update the PG14 CRs to match the new node names
  13. Update the PG14 upgrade playbook gstg inventories to match the new node names - MR: db-migration!384 (merged)
  14. Set label changecomplete /label ~change::complete

Rollback

Rollback steps - steps to be taken in the event of a need to rollback this change

Estimated Time to Complete (mins) - 2 hours

  • Revert MR - PG14 upgrade playbook gstg inventories : db-migration!384 (merged)
  • Revert - Update the PG14 CRs to match the new node names
  • IF N1 Nodes were destroyed: Revert destruction of the N1 nodes and revert default machine_type: patroni-n1-standard-8 for TF patroni-main-v14 and patroni-ci-v14 modules - Revert MR: https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/merge_requests/5548
  • IF N1 Nodes are just marked to be destroyed, revert the change https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/merge_requests/5547
    • Execute in the Main cluster 
      knife search node "roles:gstg-base-db-patroni-main-v14" -a run_list
knife node run_list remove patroni-main-v14-01-db-gstg.c.gitlab-staging-1.internal "role[gstg-base-db-patroni-to_be_destroyed]"
ssh patroni-main-v14-01-db-gstg.c.gitlab-staging-1.internal "sudo chef-client"
knife node run_list remove patroni-main-v14-02-db-gstg.c.gitlab-staging-1.internal "role[gstg-base-db-patroni-to_be_destroyed]"
ssh patroni-main-v14-02-db-gstg.c.gitlab-staging-1.internal "sudo chef-client"
knife node run_list remove patroni-main-v14-03-db-gstg.c.gitlab-staging-1.internal "role[gstg-base-db-patroni-to_be_destroyed]"
ssh patroni-main-v14-03-db-gstg.c.gitlab-staging-1.internal "sudo chef-client"
knife node run_list remove patroni-main-v14-06-db-gstg.c.gitlab-staging-1.internal "role[gstg-base-db-patroni-to_be_destroyed]"
ssh patroni-main-v14-06-db-gstg.c.gitlab-staging-1.internal "sudo chef-client"
knife node run_list remove patroni-main-v14-07-db-gstg.c.gitlab-staging-1.internal "role[gstg-base-db-patroni-to_be_destroyed]"
ssh patroni-main-v14-07-db-gstg.c.gitlab-staging-1.internal "sudo chef-client"
    • Execute for the CI cluster 
      knife search node "roles:gstg-base-db-patroni-ci-v14" -a run_list
knife node run_list remove patroni-ci-v14-01-db-gstg.c.gitlab-staging-1.internal "role[gstg-base-db-patroni-to_be_destroyed]"
ssh patroni-ci-v14-01-db-gstg.c.gitlab-staging-1.internal "sudo chef-client"
knife node run_list remove patroni-ci-v14-02-db-gstg.c.gitlab-staging-1.internal "role[gstg-base-db-patroni-to_be_destroyed]"
ssh patroni-ci-v14-02-db-gstg.c.gitlab-staging-1.internal "sudo chef-client"
knife node run_list remove patroni-ci-v14-03-db-gstg.c.gitlab-staging-1.internal "role[gstg-base-db-patroni-to_be_destroyed]"
ssh patroni-ci-v14-03-db-gstg.c.gitlab-staging-1.internal "sudo chef-client"
knife node run_list remove patroni-ci-v14-04-db-gstg.c.gitlab-staging-1.internal "role[gstg-base-db-patroni-to_be_destroyed]"
ssh patroni-ci-v14-04-db-gstg.c.gitlab-staging-1.internal "sudo chef-client"
knife node run_list remove patroni-ci-v14-05-db-gstg.c.gitlab-staging-1.internal "role[gstg-base-db-patroni-to_be_destroyed]"
ssh patroni-ci-v14-05-db-gstg.c.gitlab-staging-1.internal "sudo chef-client"
  • Revert Replacement of Backup Nodes - Revert MR: https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/merge_requests/5545
    • Execute the following in a chef-repo: 
      knife search node "roles:gstg-base-db-patroni-main-v14" -a run_list
knife node run_list remove patroni-main-v14-102-db-gstg.c.gitlab-staging-1.internal "role[gstg-base-db-patroni-main-v14-backup-replica]"
knife node run_list add patroni-main-v14-06-db-gstg.c.gitlab-staging-1.internal "role[gstg-base-db-patroni-main-v14-backup-replica]"
    • Execute the following in a chef-repo: 
      knife search node "roles:gstg-base-db-patroni-ci-v14" -a run_list
knife node run_list remove patroni-ci-v14-102-db-gstg.c.gitlab-staging-1.internal "role[gstg-base-db-patroni-ci-v14-backup-replica]"
knife node run_list add patroni-ci-v14-03-db-gstg.c.gitlab-staging-1.internal "role[gstg-base-db-patroni-ci-v14-backup-replica]"
  • Switchover the Primary Standby Leader node from node 101 BACK to node 01 on each cluster
    • Execute the following command on any cluster node, and make sure to select patroni-main-v14-01 and patroni-ci-v14-01 as candidates for the switchover; 
      sudo gitlab-patronictl switchover
    • Then check the cluster status 
      sudo gitlab-patronictl list
  • Destroy the N2 nodes - Revert MR: https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/merge_requests/5543
  • Set label changeaborted /label ~change::aborted

Monitoring

Key metrics to observe

There's no metrics from nodes in both patroni-main-v14 and patroni-ci-v14 clusters in Prometheus - https://thanos.gitlab.net/graph?g0.expr=pg_replication_lag%7Benv%3D%22gstg%22%7D&g0.tab=1&g0.stacked=0&g0.range_input=1h&g0.max_source_resolution=0s&g0.deduplicate=1&g0.partial_response=0&g0.store_matches=%5B%5D

  • Metric: Metric Name
    • Location: Dashboard URL
    • What changes to this metric should prompt a rollback: Describe Changes

Change Reviewer checklist

C4 C3 C2 C1:

  • Check if the following applies:
    • The scheduled day and time of execution of the change is appropriate.
    • The change plan is technically accurate.
    • The change plan includes estimated timing values based on previous testing.
    • The change plan includes a viable rollback plan.
    • The specified metrics/monitoring dashboards provide sufficient visibility for the change.

C2 C1:

  • Check if the following applies:
    • The complexity of the plan is appropriate for the corresponding risk of the change. (i.e. the plan contains clear details).
    • The change plan includes success measures for all steps/milestones during the execution.
    • The change adequately minimizes risk within the environment/service.
    • The performance implications of executing the change are well-understood and documented.
    • The specified metrics/monitoring dashboards provide sufficient visibility for the change.
      • If not, is it possible (or necessary) to make changes to observability platforms for added visibility?
    • The change has a primary and secondary SRE with knowledge of the details available during the change window.
    • The labels blocks deployments and/or blocks feature-flags are applied as necessary

Change Technician checklist

  • Check if all items below are complete:
    • The change plan is technically accurate.
    • This Change Issue is linked to the appropriate Issue and/or Epic
    • Change has been tested in staging and results noted in a comment on this issue.
    • A dry-run has been conducted and results noted in a comment on this issue.
    • The change execution window respects the Production Change Lock periods.
    • For C1 and C2 change issues, the change event is added to the GitLab Production calendar.
    • For C1 and C2 change issues, the SRE on-call has been informed prior to change being rolled out. (In #production channel, mention @sre-oncall and this issue and await their acknowledgement.)
    • For C1 and C2 change issues, the SRE on-call provided approval with the eoc_approved label on the issue.
    • For C1 and C2 change issues, the Infrastructure Manager provided approval with the manager_approved label on the issue.
    • Release managers have been informed (If needed! Cases include DB change) prior to change being rolled out. (In #production channel, mention @release-managers and this issue and await their acknowledgment.)
    • There are currently no active incidents that are severity1 or severity2
    • If the change involves doing maintenance on a database host, an appropriate silence targeting the host(s) should be added for the duration of the change.
Edited by Rafael Henchen