Skip to content

[GSTG] Patroni Zonal Outage Game Day

Production Change - Criticality 2 C2

Change Summary

This production issue is to be used for Gamedays as well as recovery in case of a zonal outage. It outlines the steps to be followed when adding new patroni replicas in a new zone.

Gameday execution roles and details

Role Assignee
Change Technician @thisisshreya
Change Reviewer @mattmi

Provide a brief summary indicating the affected zone

[For Gamedays only] Preparation Tasks

  1. One week before the gameday make an announcement on slack production_engineering channel. Consider also sharing this post in the appropriate environment channels, staging or production.

    • Example message:
      Next week on [DATE & TIME] we will be executing a Patroni and PGBouncer zonal outage game day. The process will involve provisioning additional capacity for these services in alternate zones. This should take approximately 90 minutes, the aim for this exercise is to test our disaster recovery capabilities and measure if we are still within our RTO & RPO targets set by the [DR working group](https://handbook.gitlab.com/handbook/company/working-groups/disaster-recovery/) for GitLab.com.
  See LINK_TO_CHANGE_ISSUE for details.

  2. Add an event to the GitLab Production calendar.

  3. Notify the release managers on Slack by mentioning @release-managers and referencing this issue and await their acknowledgment.

  4. Notify the eoc on Slack by mentioning @sre-oncall and referencing this issue and wait for approval by adding the eoc_approved label.

    • Example message:
    @release-managers or @sre-oncall LINK_TO_THIS_CR is scheduled for execution. We will be adding Patroni and PGBouncer replicas for approximately 30 minutes. Kindly review and approve the CR

  5. Post an FYI link of the slack message to the #test-platform channel on slack.

Detailed steps for the change

Change Steps - steps to take to execute the change

Execution

  1. If you are conducting a practice (Gameday) run of this, consider starting a recording of the process now.

  2. Note the start time in UTC in a comment to record this process duration.

  3. Set label changein-progress /label ~change::in-progress

  4. Create the MR to provision Patroni and PGBouncer replicas.

    ❗❗ NOTE: ❗❗ Merging the changes into master can take a while to complete, it can take up to 30 minutes before you are able to log in to the new replicas.

Validation of the Patroni replicas

  1. Wait for the instances to be built and Chef to converge.

    • Staging: Confirm that chef runs have completed for new patroni replicas it can take up to 30 minutes before they show up. A sudden increase in the y-axis of the time across the new replicas may signify that Chef has run successfully.

      Trouble shooting tips
      • Tail the serial output to confirm that the start up script executed successfully. gcloud compute --project=$project instances tail-serial-port-output $instance_name --zone=$zone --port=1 the variables $project represents the project gitlab-staging-1 for the patroni replicas, $instance_name represent the instance e.g patroni-main-v14-106-db-gstg and $zone represents the recovery zone e.g us-east1-c.
      • We could also tail bootstrap logs example: tail -f /var/tmp/bootstrap-20231108-133642.log, if we see a log that states bootstrap completed we can move on to the next steps.

  2. Once the patroni replicas are ready ssh into each replica and validate that Patroni is running:

    • sudo gitlab-patronictl list
    • Restart Patroni if not:
      • sudo systemctl enable patroni && sudo systemctl start patroni

  3. Review graphs for each of the database clusters to ensure traffic is being distributed to the new instances.

    1. Main
    2. CI
    3. Registry

  4. Validate error rates are nominal across the rails services.

    1. Web
    2. API
  • If and when successfully validated please note down the time in the comments to calculate the Gameday DR Process Time.

[For Gamedays only] Clean up

  • Set all new Patroni nodes to maintenance mode.

    • Set maintenance mode roles: 
      for node in patroni-ci-v14-{105,106} patroni-embedding-04 patroni-main-v14-{105,106} patroni-registry-v14-05; do knife node run_list add "${node}-db-gstg.c.gitlab-staging-1.internal" 'role[gstg-base-db-patroni-maintenance]' -y; done
    • Run chef-client: 
      for node in patroni-ci-v14-{105,106} patroni-embedding-04 patroni-main-v14-{105,106} patroni-registry-v14-05; do echo "${node}-db-gstg.c.gitlab-staging-1.internal" ; done | xargs -P0 -I '{}' ssh {} 'sudo chef-client'
      ❗❗NOTE❗❗ The hostnames in these commands are examples. Be sure to update them to match the newly created nodes accordingly.

  • Open and merge a MR to remove the nodes added for this game day.

    ❗❗NOTE❗❗ when removing the nodes created as part of this gameday, you may need to add the ~approve_policies label and comment atlantis approve_policies in the MR to bypass the policies before applying with atlantis apply.

Wrapping up

  • Notify the @release-managers and @sre-oncall that the exercise is complete.
  • Set label changecomplete /label ~change::complete
  • Compile the real time measurement of this process and update the Recovery Measrements Runbook.

Rollback

Rollback steps - steps to be taken in the event of a need to rollback this change

It is estimated that this will take 5m to complete

  • Notify the @release-managers and @sre-oncall that the exercise is aborted.
  • Set label changeaborted /label ~change::aborted

Monitoring

Key metrics to observe

Change Reviewer checklist

C4 C3 C2 C1:

  • Check if the following applies:
    • The scheduled day and time of execution of the change is appropriate.
    • The change plan is technically accurate.
    • The change plan includes estimated timing values based on previous testing.
    • The change plan includes a viable rollback plan.
    • The specified metrics/monitoring dashboards provide sufficient visibility for the change.

C2 C1:

  • Check if the following applies:
    • The complexity of the plan is appropriate for the corresponding risk of the change. (i.e. the plan contains clear details).
    • The change plan includes success measures for all steps/milestones during the execution.
    • The change adequately minimizes risk within the environment/service.
    • The performance implications of executing the change are well-understood and documented.
    • The specified metrics/monitoring dashboards provide sufficient visibility for the change.
      • If not, is it possible (or necessary) to make changes to observability platforms for added visibility?
    • The change has a primary and secondary SRE with knowledge of the details available during the change window.
    • The change window has been agreed upon with Release Managers in advance of the change. If the change is planned for APAC hours, this issue has an agreed pre-change approval.
    • The labels blocks deployments and/or blocks feature-flags are applied as necessary.

Change Technician checklist

  • Check if all items below are complete:
    • The change plan is technically accurate.
    • This Change Issue is linked to the appropriate Issue and/or Epic
    • Change has been tested in staging and results are noted in a comment on this issue.
    • A dry-run has been conducted and results are noted in a comment on this issue.
    • The change execution window respects the Production Change Lock periods.
    • For C1 and C2 change issues, the change event is added to the GitLab Production calendar.
    • For C1 and C2 change issues, the SRE on-call has been informed before the change is rolled out. (In the #production channel, mention @sre-oncall and this issue and await their acknowledgement.)
    • For C1 and C2 change issues, the SRE on-call provided approval with the eoc_approved label on the issue.
    • For C1 and C2 change issues, the Infrastructure Manager provided approval with the manager_approved label on the issue.
    • Release managers have been informed prior to any C1, C2, or blocks deployments change being rolled out. (In the #production channel, mention @release-managers and this issue and await their acknowledgement.)
    • There are currently no active incidents that are severity1 or severity2
    • If the change involves doing maintenance on a database host, an appropriate silence targeting the host(s) should be added for the duration of the change.
Edited by Shreya Shah