[gstg] Gitaly Zonal Outage Game Day
C2
Production Change - Criticality 2Change Summary
This production issue is to be used for Gamedays as well as recovery in case of a zonal outage. It outlines the steps to be followed when restoring Gitaly VMs in a single zone.
Gameday execution roles and details
Role | Assignee |
---|---|
Change Technician | @mattmi |
Change Technician II |
- Services Impacted - ServiceGitaly
- Time tracking - 90 minutes
- Downtime Component - 30 minutes
During this gameday, we'll be migrating the gitaly-02a nodes in us-east1-b to us-east1-d. There will be a downtime component for Gitaly as we power the original machines off to take a final snapshots for data consistency before restoring them into their new zone.
[For Gamedays only] Preparation Tasks
-
One week before the gameday make an announcement on slack production_engineering channel. Next week during the [Reliability discussions and firedrills](https://calendar.google.com/calendar/u/0/r/day/2023/12/20) meeting, we will be executing our quarterly gameday. The process will involve moving traffic away from a single zone in gstg, and moving Gitaly nodes to a new zone. This should take approximately 90 minutes, the aim for this exersice is to test our disaster recovery capabilities and measure if we are still within our RTO & RPO targets set by the [DR working group](https://handbook.gitlab.com/handbook/company/working-groups/disaster-recovery/) for GitLab.com. See https://gitlab.com/gitlab-com/gl-infra/production/-/issues/17274
-
Ensure the MRs to provision Gitaly VMs are rebased and approved. - MR that adds Gitaly servers within the available zone. Example MR
gstg
https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/merge_requests/7326gprd
https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/merge_requests/7657
- MR that adds Gitaly servers within the available zone. Example MR
-
Notify the release managers on Slack by mentioning @release-managers
and referencing this issue and await their acknowledgment. -
Notify the eoc on Slack by mentioning @sre-oncall
and referencing this issue and wait for approval by adding the eoc_approved label.- Example message:
@release-managers or @sre-oncall LINK_TO_THIS_CR is scheduled for execution. We will be taking a single Gitaly node offline for approximately 30 minutes so as to avoid dataloss. This will only affect projects that are on that Gitaly VM, it won't be used for new projects.
-
Post a similar message to the #test-platform channel on slack.
Detailed steps for the change
Change Steps - steps to take to execute the change
Execution
-
Set label changein-progress /label ~change::in-progress
-
[Skip during Gameday] Merge the MR to add a new Gitaly server within the available zone NOTE: It can take upto 20 minutes before you are able to log in to the new Gitaly server. -
Example MRs:
gstg
https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/merge_requests/7326gprd
https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/merge_requests/7657
❗ ❗ NOTE:❗ ❗ These two steps above⬆ take a while to complete the other steps can be executed in parallel. -
Example MRs:
Validation Gitaly VMs
-
Wait for the instances to be built and Chef to converge. -
Staging: Confirm that chef runs have completed for new storages and patroni replicas it can take upto 30 minutes before they show up.
Trouble shooting tips
- Tail the serial output to confirm that the start up script executed successfully.
gcloud compute --project=$project instances tail-serial-port-output $instance_name --zone=$zone --port=1
the variables$project
represents the gitaly project e.ggitaly-gstg-380a
for the gitaly storages,$instance_name
represent the instance e.ggitaly-01a-stor-gstg
, and$zone
represents the recovery zone e.gus-east1-c
. - We could also tail bootstrap logs example:
tail -f /var/tmp/bootstrap-20231108-133642.log
- Tail the serial output to confirm that the start up script executed successfully.
-
-
ssh into the Gitaly VMs: - Execute
sudo gitlab-ctl status
to validate that the servers are up - Validate that the disk is properly mounted in thanos.
❗ ❗ NOTE❗ ❗ The graph above is an example to offer guidance you may be required to change some parameters e.g.fqdn
. - Execute
POINT OF NO RETURN
-
Set the weights to zero on the affected storages. - Staging: With an admin account, navigate to Repository Storage Settings and set the weight to 0 for the affected storages.
Add the new Storages to the Rails and Gitaly configuration
-
[For Gamedays only] Stop the Gilaty node that we intend to take a snapshot. gcloud compute instances stop --project=gitlab-gitaly-gstg-380a --zone="us-east1-b" "gitaly-02a-stor-gstg"
gcloud compute snapshots create "file-gitaly-02a-gameday-snapshot" --source-disk="gitaly-02a-stor-gstg-data" --description="Part of https://gitlab.com/gitlab-com/gl-infra/production/-/issues/17991" --project=gitlab-gitaly-gstg-380a --source-disk-zone="us-east1-b"
gcloud compute instances stop --project=gitlab-gitaly-gstg-164c --zone="us-east1-b" "gitaly-02a-stor-gstg"
gcloud compute snapshots create "file-gitaly-02a-gameday-snapshot" --source-disk="gitaly-02a-stor-gstg-data" --description="Part of https://gitlab.com/gitlab-com/gl-infra/production/-/issues/17991" --project=gitlab-gitaly-gstg-164c --source-disk-zone="us-east1-b"
❗ ❗ NOTE❗ ❗ Remember to replace project names, the values used above are examples to offer guidance. -
[For Gamedays only] We can now merge the MR that adds a new Gitaly server within the available zone NOTE: It can take upto 20 minutes before you are able to log in to the new Gitaly server. -
Create an MR against chef-repo that adds the new storages to the environment Chef managed configuration. Note it will take around 30 minutes for Chef to converge. -
Create an MR against k8s-workloads/gitlab-com that will add the new storage to the K8s configuration -
MR gitlab-com/gl-infra/k8s-workloads/gitlab-com!3570 (closed)
❗ ❗ NOTE❗ ❗ ensure all pipelines have passedqa-reliable
andqa-smoke
test failures can lead to auto deploys being blocked
-
MR gitlab-com/gl-infra/k8s-workloads/gitlab-com!3570 (closed)
Validate that the new nodes are receiving traffic
-
❗ ❗ NOTE❗ ❗ The dashboards above are examples to offer guidance you may be required to change some parameters e.g.fqdn
. -
Validate a project can be created on the new Storages. -
glsh gitaly storages validate -e gstg gitaly-02-stor-gstg.c.gitlab-gitaly-gstg-164c.internal
-
glsh gitaly storages validate -e gstg gitaly-02-stor-gstg.c.gitlab-gitaly-gstg-380a.internal
-
[For Gamedays only] Clean up
-
Remove the instances that were stopped (gitaly-02a) ❗ ❗ NOTE❗ ❗ _when reverting the above⬆ MR as of this gameday you need to commentatlantis approve_policies
in the MR to bypass the policies before applying withatlantis apply
. You also need to disable the all pipelines must pass condition in the MR settings. -
Restore the weights of the affected storages.
Wrapping up
-
Notify the @release-managers
and@sre-oncall
that the exersice is complete. -
Set label changecomplete /label ~change::complete
Rollback
Rollback steps - steps to be taken in the event of a need to rollback this change
It is estimated that this will take 5m to complete
-
Notify the @release-managers
and@sre-oncall
that the exersice is aborted. -
Set label changeaborted /label ~change::aborted