Restore `patroni-01` to a useful state

Production Change - Criticality 3 C3

Change Objective Objective of the change is to restore patroni-01 to a useful state
Change Type VM instance desctruction/recreation + disk preservation
Services Impacted patroni, postgresql, consul
Change Team Members @nnelson
Change Severity Not certain, probably a ~S3 ?
Buddy check or tested in staging A colleague will review the change AND the change will be tested on staging environment
Schedule of the change 2019-09-03 2100 UTC pending any further review of change plan
Duration of the change An hour or two
Detailed steps for the change. Each step must include: See below

"The patroni-01 node is ready to be rebuilt." -- #1061 (comment 210008102)

Some useful reference documentation for Patroni member management:

https://gitlab.com/gitlab-com/runbooks/blob/master/howto/patroni-management.md#scaling-the-cluster-up

Summary of steps

  • Verify that patroni-01 is indeed receiving a replication stream.
  • Stop patroni-01 from participating in streaming replication
  • Shut down the patroni-01 system
  • Clone the patroni-01-db-gprd-data disk
  • Clone the patroni-01-db-gprd-log disk
  • Delete the patroni-01 virtual machine instance
  • Delete the patroni-01-db-gprd-data disk
  • Delete the patroni-01-db-gprd-log disk
  • Re-create the patroni-01 virtual machine instance and disk
  • Start the patroni service on the new patroni-01 system

Stop patroni-01 from participating in streaming replication

The streaming replication protocol command might not work. (https://www.postgresql.org/docs/current/protocol-replication.html)

Noted here for completeness, but don't invoke:

# gitlab-psql --command "DROP_REPLICATION_SLOT 'patroni_01_db_gprd_c_gitlab_production_internal' WAIT;"

  • pre-conditions for execution of the step

  • Verify that patroni-01 is indeed receiving a replication stream.

ssh patroni-01-db-gprd.c.gitlab-production.internal
ps -eaf | grep "wal receiver process" | grep -v grep

Should print something like:

gitlab-+ 57995 57863  7 Aug27 ?        03:37:42 postgres: pg-ha-cluster: wal receiver process   streaming 1328D/84853630

Also:

sudo ls -lat /var/opt/gitlab/postgresql/data/pg_xlog | head -n2

Should print something like:

total 8963432
-rw-------  1 gitlab-psql gitlab-psql 16777216 Aug 29 17:58 0000001A0001328E0000000B

Furthermore, shell into the leader patroni node (currently patroni-10-db-gprd.c.gitlab-production.internal) and execute the following commands:

ssh patroni-10-db-gprd.c.gitlab-production.internal
sudo gitlab-psql --tuples-only --command "SELECT * from pg_replication_slots where slot_name = 'patroni_01_db_gprd_c_gitlab_production_internal'"

Should print something like:

 patroni_01_db_gprd_c_gitlab_production_internal |        | physical  |        |          | t      |      56418 | 3573786175 |              | 13292/5E8F9F40 |

  • execution commands for the step

  • Stop patroni-01 from participating in streaming replication

sudo gitlab-psql --command "SELECT pg_drop_replication_slot('patroni_01_db_gprd_c_gitlab_production_internal');"
  • post-execution validation for the step

Should be able to ust re-run the command from the pre-execution step and verify its output no longer includes results for the patroni_01 query.

ssh patroni-10-db-gprd.c.gitlab-production.internal
sudo gitlab-psql --tuples-only --command "SELECT * from pg_replication_slots where slot_name = 'patroni_01_db_gprd_c_gitlab_production_internal'"

Should print nothing.

  • rollback of the step

Rolling back from a DROP would be to CREATE, so:

sudo gitlab-psql --command "SELECT pg_create_physical_replication_slot('patroni_01_db_gprd_c_gitlab_production_internal');"

But, it is unclear whether this would be necessary or desired.

Shut-down the system of the patroni-01 virtual machine instance

  • Shut down the patroni-01 system

Note: Operations such as this one may be more easily executed in the Google Cloud Console.

  • pre-conditions for execution of the step

  • execution commands for the step

Navigate to: https://console.cloud.google.com/compute/instancesDetail/zones/us-east1-c/instances/patroni-01-db-gprd?project=gitlab-production

Smash that "Stop" button

  • post-execution validation for the step

  • rollback of the step

That "Stop" button might toggle to "Start", so probably click that.

Make an archive copy of the patroni-01-db-gprd-data disk

  • Clone the patroni-01-db-gprd-data disk

Here the data disk must be cloned with a unique name indicating its archival because the original data disk here must be deleted in order to allow its re-building via terraform and re-use of the same name: "patroni-01-db-gprd-data".

  • pre-conditions for execution of the step

Ensure that the patroni-01 instance has stopped.

  • execution commands for the step

Navigate to: https://console.cloud.google.com/compute/instancesDetail/zones/us-east1-c/instances/patroni-01-db-gprd?project=gitlab-production

Scroll down to the "Additional disks" section, and click the "patroni-01-db-gprd-data" link.

Click the "Create Image" button at the top of the page.

Enter "archive-patroni-01-db-gprd-data-2019-09-03" into the "Name" field.

In the "Location" section, click the "Regional" radio button.

Ensure that us-east1 (South Carolina) (default) is selected.

Click the "Create" button near the bottom of the page.

  • post-execution validation for the step

Wait until the image has finished being created.

TODO Decide whether or not to attach the disk to some other instance and mount its filesystem in the instance system for validation purposes.

  • rollback of the step

Delete the cloned image, but not the original image.

Make an archive copy of the patroni-01-db-gprd-data disk

  • Clone the patroni-01-db-gprd-log disk

Repeat the steps from the previous section, but replace patroni-01-db-gprd-data with patroni-01-db-gprd-log.

Delete patroni-01 virtual machine instance

  • Delete the patroni-01 virtual machine instance

Note: Operations such as this one may be more easily executed in the Google Cloud Console.

  • pre-conditions for execution of the step

Execution of the previous steps. Ensure that the instance system has stopped.

  • execution commands for the step

Navigate to: https://console.cloud.google.com/compute/instancesDetail/zones/us-east1-c/instances/patroni-01-db-gprd?project=gitlab-production

Click the "Delete" button.

Confirm appropriately.

  • post-execution validation for the step

Invoking a terraform plan for the patroni module will produce change plans to re-create a single node called patroni-01.

user@workstation:~/src/gitlab-com-infrastructure[master]$ terraform plan -target module.patroni
  • rollback of the step

Repeat the above steps except for disk image clone and this one.

Delete patroni-01-db-gprd-data disk

  • Delete the patroni-01-db-gprd-data disk

Note: Operations such as this one may be more easily executed in the Google Cloud Console.

  • pre-conditions for execution of the step

Double-check to make sure that the instance deletion step has not also deleted the disk images.

  • execution commands for the step

Navigate to: https://console.cloud.google.com/compute/disksDetail/zones/us-east1-c/disks/patroni-01-db-gprd-data?project=gitlab-production

Click the "Delete" button.

Confirm appropriately.

  • post-execution validation for the step

Navigate to: https://console.cloud.google.com/compute/disksDetail/zones/us-east1-c/disks/patroni-01-db-gprd-data?project=gitlab-production

Confirm that the Google Cloud Console cannot find any such disk.

Can't fetch the requested data right now. Try again in a few minutes.
  • rollback of the step

One would have to rename the appropriate cloned archival disk to patroni-01-db-gprd-data, I suppose.

Delete patroni-01-db-gprd-log disk

  • Delete the patroni-01-db-gprd-log disk

Repeat the steps from the section above, but replace patroni-01-db-gprd-data with patroni-01-db-gprd-log.

Recreate patroni-01 with terraform

  • Create the patroni-01 virtual machine instance and disk

  • pre-conditions for execution of the step

Invoking a terraform plan for the patroni module will produce change plans to re-create a single node called patroni-01.

user@workstation:~$ cd ~/src/gitlab-com-infrastructure
user@workstation:~/src/gitlab-com-infrastructure[master]$ terraform plan -target module.patroni
  • execution commands for the step

First examine the terraform plan.

user@workstation:~/src/gitlab-com-infrastructure[master]$ tf plan -target module.patroni

Verify that the plan is to create a new node.

Bring the node back as a new replica.

user@workstation:~/src/gitlab-com-infrastructure[master]$ terraform apply -target module.patroni
  • post-execution validation for the step

One should be able to establish a shell session to the system of the new virtual machine instance with the same fqdn of the original node.

ssh patroni-01-db-gprd.c.gitlab-production.internal
  • rollback of the step

Not necessary.

Start patroni

  • Start the patroni service on the new patroni-01 system

  • pre-conditions for execution of the step

On any other Patroni member, list all the cluster members and their statuses.

sudo gitlab-patronictl list

Ensure that the other cluster members are identical to those seen by running the same command on another cluster member.

  • execution commands for the step
systemctl enable patroni && systemctl start patroni
  • post-execution validation for the step

Follow the patroni logs. A pg_basebackup will take several hours, after which point streaming replication will begin. Silence alerts as necessary.

Run gitlab-patronictl list to check the state of the patroni cluster, you should see the new node join the cluster and go through the following states:

  • creating replica
  • starting
  • running

Overall replica progress may be monitored by comparing the disk usage of a known replicant with a complete set to the disk usage of our new replica:

$ ssh patroni-04-db-gprd.c.gitlab-production.internal 'sudo du -s /var/opt/gitlab/postgresql/data/'
4097909648      /var/opt/gitlab/postgresql/data/
$ ssh patroni-01-db-gprd.c.gitlab-production.internal 'sudo du -s /var/opt/gitlab/postgresql/data/'
1956051240      /var/opt/gitlab/postgresql/data/
  • rollback of the step

Not necessary.

Edited by Nels Nelson