Skip to content

[GSTG] Zonal Gitaly Restore/Test Gameday

Production Change - Criticality 2 C2

Change Summary

This production issue is to be used for Gamedays as well as recovery in case of a zonal outage. It outlines the steps to be followed when restoring Gitaly VMs in a single zone.

This production issue is divided into two sections , one for planned gamedays and one for in the event of an actual outage

Execution roles and details

Role Assignee
Change Technician @swainaina
Change Reviewer @cmcfarland

  • Services Impacted - ServiceGitaly

  • Time tracking -

    90 minutes

  • Downtime Component -

    30 minutes

Provide a brief summary indicating the affected zone

We'll be restoring gitaly-02 in us-east1-b as gitaly-02a in us-east1-c. gitaly-01a will be left in us-east1-b, as currently all GSTG Gitaly nodes are in the same zone.

Planned Game Day (GSTG)

Planned Game Day (GSTG)

Preparation Tasks

The Week Prior

  1. One week before the gameday make an announcement on slack production_engineering channel and request approval from @release-managers. Additionally, share the message in staging and test-platform channels.

      Next week on [DATE & TIME] we will be executing a Gitaly gameday. The process will involve moving traffic away from a single zone in gstg, and moving
  Gitaly nodes to a new zone. This should take approximately 90 minutes, the aim for this exercise is to test our disaster recovery capabilities and measure if we are still within our RTO & RPO targets set by the [DR working group](https://handbook.gitlab.com/handbook/company/working-groups/disaster-recovery/) for GitLab.com.
  See https://gitlab.com/gitlab-com/gl-infra/production/-/issues/17274

  2. Notify the release managers on Slack by mentioning @release-managers and referencing this issue and await their acknowledgment.

  3. Add an event to the GitLab Production calendar.

  4. Ensure you have PAT, you would require admin API access , please select api , read_api and admin_mode scopes , also validate if you have SSH key setup for staging.

Day Of

  1. Notify the release managers on Slack by mentioning @release-managers and referencing this issue and await their acknowledgment.

  2. Notify the eoc on Slack by mentioning @sre-oncall and referencing this issue and wait for approval by adding the eoc_approved label.

    • Example message:
    @release-managers or @sre-oncall LINK_TO_THIS_CR is scheduled for execution.
We will be taking a single Gitaly node offline for approximately 30 minutes so as to avoid dataloss.
This will only affect projects that are on that Gitaly VM, it won't be used for new projects. Kindly review and approve the CR

  3. Share this message in the #test-platform and #staging channels on slack.

Detailed steps for the change

Change Steps

Execution

  1. Consider starting a recording of the process now.

  2. Note the start time in UTC in a comment to record this process duration.

  3. Note down the md5sum of a repository on the affected storage in a comment on this issue.

    • glsh gitaly repositories checksum -s gitaly-02-stor-gstg.c.gitlab-gitaly-gstg-164c.internal -e gstg -g gitlab-com
    • replace gitaly-02-stor-gstg.c.gitlab-gitaly-gstg-164c.internal with the storage name (not necessarily fqdn) of one of the impacted storage.
    • <name>-stor-gstg.c.gitlab-gitaly-gstg-164c.internal is likely to be the storage name for a host, you can cross-reference the rails configuration for validation , additionally assuming we know which zone is affected , we can navigate to gitaly-multi-project.tf to validate.
    • Note that there are multiple Gitaly projects in gstg and gprd , in the above example we are calculating the md5sum of the one of the impacted storage ie: from the gitlab-gitaly-gstg-164c project. To get a list of gitaly projects execute the following
    gcloud projects list --filter="name:gitlab-gitaly-<environment>*"

  4. Use this script to create to push commits to a new branch in config-mgmt , gitlab-com and chef-repo

    cd runbooks/scripts/disaster-recovery
bundle
bundle exec ruby gitaly-replace-nodes.rb -e gstg -z us-east1-d --app-config --commit --push --node-to-replace=gitaly-02-stor-gstg
# Where all of the Gitaly nodes in us-east1-d will be redistributed among the remaining zones.

  5. Create merge requests in config-mgmt , gitlab-com and chef-repo against the new branches created by the script, for example

  1. Set label changein-progress /label ~change::in-progress

  2. Set the weights to zero on the affected storages.

    • With an admin account, navigate to Repository Storage Settings and set the weight to 0 for the affected storages that correspond to the instances being replaced.

  3. Stop the Gitaly nodes and create new snapshots. ( Since we currently have two Gitaly projects in gstg)

    :exclamation::exclamation: NOTE: :exclamation::exclamation: The zone, instance and disk names in these commands are examples. They need to be modified to match the values corresponding to the instances you are replacing.

gcloud compute instances stop --project=gitlab-gitaly-gstg-380a --zone="us-east1-b" "gitaly-01-stor-gstg"

gcloud compute snapshots create "file-gitaly-01-gameday-snapshot" --source-disk="gitaly-01-stor-gstg-data" --description="Part of https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18627" --project=gitlab-gitaly-gstg-380a --source-disk-zone="us-east1-b"

gcloud compute instances stop --project=gitlab-gitaly-gstg-164c --zone="us-east1-b" "gitaly-01-stor-gstg"

gcloud compute snapshots create "file-gitaly-01-gameday-snapshot" --source-disk="gitaly-01-stor-gstg-data" --description="Part of https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18627" --project=gitlab-gitaly-gstg-164c --source-disk-zone="us-east1-b"
  1. Verify that the repositories on the storage are no longer available and paste the output as a comment. The same glsh gitaly repositories checksum command from above should exit with an error.
    • glsh gitaly repositories checksum -s gitaly-02-stor-gstg.c.gitlab-gitaly-gstg-164c.internal -e gstg -g gitlab-com
  2. Validate that the terraform plan is creating disks using the snapshots that were just taken in the previous step.
    • Review the plan and identify the source snapshot for the Data disks and that the snapshots are the snapshot created earlier in this process.
    • Merge the config-mgmt MR to provision the new Gitaly instances if things appear correct.
    • :exclamation::exclamation: NOTE: :exclamation::exclamation: This step above ⬆ will take 10-20 minutes to complete

Validation Gitaly VMs

  1. Wait for the instances to be built and Chef to converge.

    • Confirm that chef runs have completed for new storages, it can take up to 20 minutes before they show up.

    • Validate if Bootstrapping is completed

      • We could also tail bootstrap the latest logs example: tail -f /var/tmp/bootstrap-20231108-133642.log,20231108 represents the date of creation of the log file in yyyy-mm-dd format, 133642 represents time in UTC in HH-MM-SSformat.
      • Bootstrapping has completed once the machine has rebooted, and the most recent bootstrap script log from the game day has reported: Bootstrap finished
      Trouble shooting tips
      • Tail the serial output to confirm that the start up script executed successfully. gcloud compute --project=$project instances tail-serial-port-output $instance_name --zone=$zone --port=1 the variables $project represents the gitaly project e.g gitaly-gstg-380a for the gitaly storages, $instance_name represent the instance e.g gitaly-01a-stor-gstg, and $zone represents the recovery zone e.g us-east1-c.

  2. ssh into the Gitaly VMs:

    • Ensure a separate disk has been mounted to /var/opt/gitlab
      • shell: mount | grep /opt/gitlab
      • Mimir :exclamation::exclamation:NOTE:exclamation::exclamation: The graph above is an example to offer guidance you may be required to change some parameters e.g. fqdn.
    • Execute sudo gitlab-ctl status to validate that the servers are up

POINT OF NO RETURN

Add the new Storages to the Rails and Gitaly configuration

  1. Merge the k8s-workload/gitlab-com MR
  2. Merge the chef-repo MR
  3. Once the chef-repo pipeline succeeds, force a chef-client run across the entire Gitaly fleet.
    • knife ssh -t 3 -C 10 'chef_environment:gstg AND (name:gitaly* OR name:file-hdd*)' 'sudo chef-client'

Validate that the new nodes are receiving traffic

  1. https://dashboards.gitlab.net/d/gitaly-host-detail/gitaly3a-host-detail?orgId=1&var-PROMETHEUS_DS=mimir-gitlab-gstg&var-environment=gstg&var-fqdn=gitaly-01-stor-gstg.c.gitlab-gitaly-gstg-164c.internal&viewPanel=1687078843

  2. https://dashboards.gitlab.net/d/gitaly-host-detail/gitaly3a-host-detail?orgId=1&var-PROMETHEUS_DS=mimir-gitlab-gstg&var-environment=gstg&var-fqdn=gitaly-01-stor-gstg.c.gitlab-gitaly-gstg-380a.internal&viewPanel=1687078843

    :exclamation::exclamation:NOTE:exclamation::exclamation: The dashboards above are examples to offer guidance you may be required to change some parameters e.g. fqdn.

  3. Validate a project can be created on the new Storages.

    • glsh gitaly storages validate -e gstg gitaly-01-stor-gstg.c.gitlab-gitaly-gstg-164c.internal
    • glsh gitaly storages validate -e gstg gitaly-01-stor-gstg.c.gitlab-gitaly-gstg-380a.internal

    :exclamation::exclamation:NOTE:exclamation::exclamation: Please note it may take several minutes for validation commands to give reliable output.

  4. Validate the repository we previously collected an md5sum from is available, and the checksums match. Copy the md5sum line as a comment in this issue.

    • glsh gitaly repositories checksum -s gitaly-02-stor-gstg.c.gitlab-gitaly-gstg-164c.internal -e gstg -g gitlab-com

    :exclamation::exclamation:NOTE:exclamation::exclamation: Remember to replace the storage name with the one that is having hosts migrated for. The values used above are examples to offer guidance.

  5. Validate that the number of errors returned by the application are at a nominal level.

  6. Restore the weights of the affected storages.

  7. Note the conclusion time in UTC in a comment to record this process duration.

Clean up

  • Create and merge an MR to the config-mgmt repo, replacing the now unused Gitaly instances with our newly created Gitaly nodes in gitaly-multi-project.tf.

    • If the gitaly node replacement ruby script was used, ensure `gitaly-recovery-nodes.tf has the original contents.

      :exclamation::exclamation:NOTE:exclamation::exclamation: When deleting Gitaly instances, you need to add the ~approve-policies label and comment atlantis approve_policies in the MR to bypass the policies before applying with atlantis apply.

  • Remove any data disk snapshots that were taken before restoring the nodes. These will not be removed automatically.

Wrapping up

  • Notify the @release-managers and @sre-oncall that the exercise is complete.

    :exclamation::exclamation:NOTE:exclamation::exclamation: Ensure all unused Gitaly nodes have been deleted before signalling completion.

  • Set label changecomplete /label ~change::complete

  • Compile the real time measurement of this process and update the Recovery Measurements Runbook.

Rollback

Rollback steps - steps to be taken in the event of a need to rollback this change

If the point of no return has not been passed:

Monitoring

Key metrics to observe

Change Reviewer checklist

C4 C3 C2 C1:

  • Check if the following applies:
    • The scheduled day and time of execution of the change is appropriate.
    • The change plan is technically accurate.
    • The change plan includes estimated timing values based on previous testing.
    • The change plan includes a viable rollback plan.
    • The specified metrics/monitoring dashboards provide sufficient visibility for the change.

C2 C1:

  • Check if the following applies:
    • The complexity of the plan is appropriate for the corresponding risk of the change. (i.e. the plan contains clear details).
    • The change plan includes success measures for all steps/milestones during the execution.
    • The change adequately minimizes risk within the environment/service.
    • The performance implications of executing the change are well-understood and documented.
    • The specified metrics/monitoring dashboards provide sufficient visibility for the change.
      • If not, is it possible (or necessary) to make changes to observability platforms for added visibility?
    • The change has a primary and secondary SRE with knowledge of the details available during the change window.
    • The change window has been agreed upon with Release Managers in advance of the change. If the change is planned for APAC hours, this issue has an agreed pre-change approval.
    • The labels blocks deployments and/or blocks feature-flags are applied as necessary.

Change Technician checklist

  • Check if all items below are complete:
    • The change plan is technically accurate.
    • This Change Issue is linked to the appropriate Issue and/or Epic
    • Change has been tested in staging and results are noted in a comment on this issue.
    • A dry-run has been conducted and results are noted in a comment on this issue.
    • The change execution window respects the Production Change Lock periods.
    • For C1 and C2 change issues, the change event is added to the GitLab Production calendar.
    • For C1 and C2 change issues, the SRE on-call has been informed before the change is rolled out. (In the #production channel, mention @sre-oncall and this issue and await their acknowledgement.)
    • For C1 and C2 change issues, the SRE on-call provided approval with the eoc_approved label on the issue.
    • For C1 and C2 change issues, the Infrastructure Manager provided approval with the manager_approved label on the issue.
    • Release managers have been informed prior to any C1, C2, or blocks deployments change being rolled out. (In the #production channel, mention @release-managers and this issue and await their acknowledgement.)
    • There are currently no active incidents that are severity1 or severity2
    • If the change involves doing maintenance on a database host, an appropriate silence targeting the host(s) should be added for the duration of the change.
Edited by Silvester Wainaina