Skip to content

[GSTG] Zonal Gitaly Restore/Test Gameday

Production Change - Criticality 2 C2

Change Summary

This production issue is to be used for Gamedays as well as recovery in case of a zonal outage. It outlines the steps to be followed when restoring Gitaly VMs in a single zone.

Gameday execution roles and details

Role Assignee
Change Technician @thisisshreya
Change Reviewer @mattmi / @cmcfarland
  • Services Impacted - ServiceGitaly
  • Time tracking - 90 minutes
  • Downtime Component - 30 minutes

During this game day, we'll be restoring a gitaly nodes in us-east1-c to us-east1-d (replacing gitaly-01a)

Production Outage

Perform these steps in the event of an outage in production

❗NOTE❗❗ These steps are intended to be run in the event that there is a zonal outage, and the set of Gitaly VMs in that zone are completely unavailable.

Preparation Tasks

  1. Notify the release managers on Slack by mentioning @release-managers and referencing this issue and await their acknowledgment.
  2. Notify the EOC on Slack by mentioning @sre-oncall and referencing this issue and wait for approval by adding the eoc_approved label.
  3. Post a notification in the production Slack channel
  4. Ensure all merge requests have been rebased if necessary and approved.

Detailed steps for the change

Change Steps

Execution

  1. Set label changein-progress /label ~change::in-progress
  2. Note the start time in UTC in a comment to record this process duration.
  3. Prepare merge requests ❗❗NOTE❗❗ Merge requests should target a single zonal outage recovery branch, to be determined and created beforehand. NOT master
  4. Merge the config-mgmt MR to provision the new Gitaly instances into the zonal outage recovery branch.
  5. Wait for the main zonal outage recovery branch to be merged into master.

Validation Gitaly VMs

  1. Wait for the instances to be built and Chef to converge.

    • Confirm that chef runs have completed for new storages, it can take up to 20 minutes before they show up.

      Trouble shooting tips
      • Tail the serial output to confirm that the start up script executed successfully. gcloud compute --project=$project instances tail-serial-port-output $instance_name --zone=$zone --port=1 the variables $project represents the gitaly project e.g gitlab-gitaly-gprd-380a for the gitaly storages, $instance_name represent the instance e.g gitaly-01a-stor-gstg, and $zone represents the recovery zone e.g us-east1-c.
      • We could also tail bootstrap logs example: tail -f /var/tmp/bootstrap*.log
      • Bootstrapping has completed once the machine has rebooted, and the most recent bootstrap script log from the game day has reported: Bootstrap finished
  2. ssh into the Gitaly VMs:
    • Ensure a separate disk has been mounted to /var/opt/gitlab
    • Execute sudo gitlab-ctl status to validate that the servers are up
    • Validate that the data disk is properly mounted:

      • shell: mount | grep /opt/gitlab

      • Mimir

        ❗❗NOTE❗❗ The graph above is an example to offer guidance you may be required to change some parameters e.g. fqdn.

POINT OF NO RETURN

Add the new Storages to the Rails and Gitaly configuration

  1. Merge the k8s-workload/gitlab-com MR
  2. Merge the chef-repo MR
  3. Once the chef-repo pipeline succeeds, force a chef-client run across the entire Gitaly fleet.
    • knife ssh -t 3 -C 50 'chef_environment:gprd AND (name:gitaly* OR name:file-hdd*)' 'sudo chef-client'

Validate that the new nodes are receiving traffic

  1. https://dashboards.gitlab.net/d/gitaly-host-detail/gitaly3a-host-detail?orgId=1&var-PROMETHEUS_DS=PA258B30F88C30650&var-environment=gstg&var-fqdn=gitaly-01a-stor-gstg.c.gitlab-gitaly-gstg-164c.internal&viewPanel=1687078843

  2. https://dashboards.gitlab.net/d/gitaly-host-detail/gitaly3a-host-detail?orgId=1&var-PROMETHEUS_DS=PA258B30F88C30650&var-environment=gstg&var-fqdn=gitaly-01a-stor-gstg.c.gitlab-gitaly-gstg-380a.internal&viewPanel=1687078843

    ❗❗NOTE❗❗ The dashboards above are examples to offer guidance you may be required to change some parameters e.g. fqdn.

  3. Validate a project can be created on the new Storages.

    • glsh gitaly storages validate -e gstg gitaly-01-stor-gstg.c.gitlab-gitaly-gstg-164c.internal
    • glsh gitaly storages validate -e gstg gitaly-01-stor-gstg.c.gitlab-gitaly-gstg-380a.internal

    ❗❗NOTE❗❗ Remember to replace the storage name with the one that is having hosts migrated for. The values used above are examples to offer guidance.

  4. Note the conclusion time in UTC in a comment to record this process duration.

Wrapping Up

Planned Game Day (GSTG)

Preparation Tasks

The Week Prior

  1. One week before the gameday make an announcement on slack production_engineering channel and request approval from @release-managers. Additionally, share the message in staging and test-platform channels.

      Next week on [DATE & TIME] we will be executing a Gitaly gameday. The process will involve moving traffic away from a single zone in gstg, and moving 
  Gitaly nodes to a new zone. This should take approximately 90 minutes, the aim for this exercise is to test our disaster recovery capabilities and measure if we are still within our RTO & RPO targets set by the [DR working group](https://handbook.gitlab.com/handbook/company/working-groups/disaster-recovery/) for GitLab.com.
  See https://gitlab.com/gitlab-com/gl-infra/production/-/issues/17274

  2. Notify the release managers on Slack by mentioning @release-managers and referencing this issue and await their acknowledgment.

  3. Add an event to the GitLab Production calendar.

Day Of

  1. Notify the release managers on Slack by mentioning @release-managers and referencing this issue and await their acknowledgment.

  2. Notify the eoc on Slack by mentioning @sre-oncall and referencing this issue and wait for approval by adding the eoc_approved label.

    • Example message:
    @release-managers or @sre-oncall LINK_TO_THIS_CR is scheduled for execution. 
We will be taking a single Gitaly node offline for approximately 30 minutes so as to avoid dataloss. 
This will only affect projects that are on that Gitaly VM, it won't be used for new projects. Kindly review and approve the CR

  3. Share this message in the #test-platform and #staging channels on slack.

Detailed steps for the change

Change Steps

Execution

  1. Consider starting a recording of the process now.
  2. Note the start time in UTC in a comment to record this process duration.
  3. Set label changein-progress /label ~change::in-progress
  4. Note down the md5sum of a repository on the affected storage in a comment on this issue.
    • glsh gitaly repositories checksum -s gitaly-02-stor-gstg.c.gitlab-gitaly-gstg-164c.internal -e gstg -g gitlab-com
    • replace gitaly-02-stor-gstg.c.gitlab-gitaly-gstg-164c.internal with the storage name (not necessarily fqdn) of one of the impacted storage.
  5. Prepare merge requests
  1. Set the weights to zero on the affected storages.
    • With an admin account, navigate to Repository Storage Settings and set the weight to 0 for the affected storages that correspond to the instances being replaced.
  2. Stop the Gitaly nodes and create new snapshots.

    • gcloud compute instances stop --project=gitlab-gitaly-gstg-380a --zone="us-east1-b" "gitaly-01-stor-gstg"

    • gcloud compute snapshots create "file-gitaly-01-gameday-snapshot" --source-disk="gitaly-01-stor-gstg-data" --description="Part of https://gitlab.com/gitlab-com/gl-infra/production/-/issues/17189" --project=gitlab-gitaly-gstg-380a --source-disk-zone="us-east1-b"

    • gcloud compute instances stop --project=gitlab-gitaly-gstg-164c --zone="us-east1-b" "gitaly-01-stor-gstg"

    • gcloud compute snapshots create "file-gitaly-01-gameday-snapshot" --source-disk="gitaly-01-stor-gstg-data" --description="Part of https://gitlab.com/gitlab-com/gl-infra/production/-/issues/17189" --project=gitlab-gitaly-gstg-164c --source-disk-zone="us-east1-b"

      ❗❗ NOTE: ❗❗ The zone, instance and disk names in these commands are examples. They need to be modified to match the values corresponding to the instances you are replacing.

  3. Verify that the repositories on the storage are no longer available and paste the output as a comment. The same glsh gitaly repositories checksum command from above should exit with an error.
    • glsh gitaly repositories checksum -s gitaly-02-stor-gstg.c.gitlab-gitaly-gstg-164c.internal -e gstg -g gitlab-com
  4. Merge the config-mgmt MR to provision the new Gitaly instances.
    • Validate that the terraform plan is creating disks using the snapshots that were just taken in the previous step.
    • ❗❗ NOTE: ❗❗ This step above ⬆ will take 10-20 minutes to complete

Validation Gitaly VMs

  1. Wait for the instances to be built and Chef to converge.

    • Confirm that chef runs have completed for new storages, it can take up to 20 minutes before they show up.

      Trouble shooting tips
      • Tail the serial output to confirm that the start up script executed successfully. gcloud compute --project=$project instances tail-serial-port-output $instance_name --zone=$zone --port=1 the variables $project represents the gitaly project e.g gitaly-gstg-380a for the gitaly storages, $instance_name represent the instance e.g gitaly-01a-stor-gstg, and $zone represents the recovery zone e.g us-east1-c.
      • We could also tail bootstrap logs example: tail -f /var/tmp/bootstrap-20231108-133642.log
      • Bootstrapping has completed once the machine has rebooted, and the most recent bootstrap script log from the game day has reported: Bootstrap finished
  2. ssh into the Gitaly VMs:
    • Ensure a separate disk has been mounted to /var/opt/gitlab
    • Execute sudo gitlab-ctl status to validate that the servers are up
    • Validate that the data disk is properly mounted:

      • shell: mount | grep /opt/gitlab

      • Mimir

        ❗❗NOTE❗❗ The graph above is an example to offer guidance you may be required to change some parameters e.g. fqdn.

POINT OF NO RETURN

Add the new Storages to the Rails and Gitaly configuration

  1. Merge the k8s-workload/gitlab-com MR
  2. Merge the chef-repo MR
  3. Once the chef-repo pipeline succeeds, force a chef-client run across the entire Gitaly fleet.
    • knife ssh -t 3 -C 10 'chef_environment:gstg AND (name:gitaly* OR name:file-hdd*)' 'sudo chef-client'

Validate that the new nodes are receiving traffic

  1. https://dashboards.gitlab.net/d/gitaly-host-detail/gitaly3a-host-detail?orgId=1&var-PROMETHEUS_DS=PA258B30F88C30650&var-environment=gstg&var-fqdn=gitaly-01a-stor-gstg.c.gitlab-gitaly-gstg-164c.internal&viewPanel=1687078843

  2. https://dashboards.gitlab.net/d/gitaly-host-detail/gitaly3a-host-detail?orgId=1&var-PROMETHEUS_DS=PA258B30F88C30650&var-environment=gstg&var-fqdn=gitaly-01a-stor-gstg.c.gitlab-gitaly-gstg-380a.internal&viewPanel=1687078843

    ❗❗NOTE❗❗ The dashboards above are examples to offer guidance you may be required to change some parameters e.g. fqdn.

  3. Validate a project can be created on the new Storages.

    • glsh gitaly storages validate -e gstg gitaly-01-stor-gstg.c.gitlab-gitaly-gstg-164c.internal
    • glsh gitaly storages validate -e gstg gitaly-01-stor-gstg.c.gitlab-gitaly-gstg-380a.internal

  4. Validate the repository we previously collected an md5sum from is available, and the checksums match. Copy the md5sum line as a comment in this issue.

    • glsh gitaly repositories checksum -s gitaly-02-stor-gstg.c.gitlab-gitaly-gstg-164c.internal -e gstg -g gitlab-com

    ❗❗NOTE❗❗ Remember to replace the storage name with the one that is having hosts migrated for. The values used above are examples to offer guidance.

  5. Validate that the number of errors returned by the application are at a nominal level.

  6. Restore the weights of the affected storages.

  7. Note the conclusion time in UTC in a comment to record this process duration.

Clean up

  • Create and merge a MR to the config-mgmt repo, removing the now unused Gitaly instances

    ❗❗NOTE❗❗ when deleting Gitaly instances, you need to add the ~approve-policies label and comment atlantis approve_policies in the MR to bypass the policies before applying with atlantis apply.

  • Remove any data disk snapshots that were taken prior to restoring the nodes. These will not be removed automatically.

Wrapping up

  • Notify the @release-managers and @sre-oncall that the exercise is complete.

    ❗❗NOTE❗❗ Ensure all unused Gitaly nodes have been deleted prior to signaling completion.

  • Set label changecomplete /label ~change::complete

  • Compile the real time measurement of this process and update the Recovery Measrements Runbook.

Rollback

Rollback steps - steps to be taken in the event of a need to rollback this change

If the point of no return has not been passed:

Monitoring

Key metrics to observe

Change Reviewer checklist

C4 C3 C2 C1:

  • Check if the following applies:
    • The scheduled day and time of execution of the change is appropriate.
    • The change plan is technically accurate.
    • The change plan includes estimated timing values based on previous testing.
    • The change plan includes a viable rollback plan.
    • The specified metrics/monitoring dashboards provide sufficient visibility for the change.

C2 C1:

  • Check if the following applies:
    • The complexity of the plan is appropriate for the corresponding risk of the change. (i.e. the plan contains clear details).
    • The change plan includes success measures for all steps/milestones during the execution.
    • The change adequately minimizes risk within the environment/service.
    • The performance implications of executing the change are well-understood and documented.
    • The specified metrics/monitoring dashboards provide sufficient visibility for the change.
      • If not, is it possible (or necessary) to make changes to observability platforms for added visibility?
    • The change has a primary and secondary SRE with knowledge of the details available during the change window.
    • The change window has been agreed upon with Release Managers in advance of the change. If the change is planned for APAC hours, this issue has an agreed pre-change approval.
    • The labels blocks deployments and/or blocks feature-flags are applied as necessary.

Change Technician checklist

  • Check if all items below are complete:
    • The change plan is technically accurate.
    • This Change Issue is linked to the appropriate Issue and/or Epic
    • Change has been tested in staging and results are noted in a comment on this issue.
    • A dry-run has been conducted and results are noted in a comment on this issue.
    • The change execution window respects the Production Change Lock periods.
    • For C1 and C2 change issues, the change event is added to the GitLab Production calendar.
    • For C1 and C2 change issues, the SRE on-call has been informed before the change is rolled out. (In the #production channel, mention @sre-oncall and this issue and await their acknowledgement.)
    • For C1 and C2 change issues, the SRE on-call provided approval with the eoc_approved label on the issue.
    • For C1 and C2 change issues, the Infrastructure Manager provided approval with the manager_approved label on the issue.
    • Release managers have been informed prior to any C1, C2, or blocks deployments change being rolled out. (In the #production channel, mention @release-managers and this issue and await their acknowledgement.)
    • There are currently no active incidents that are severity1 or severity2
    • If the change involves doing maintenance on a database host, an appropriate silence targeting the host(s) should be added for the duration of the change.
Edited by Shreya Shah