Skip to content

[gstg] HAProxy Zonal Outage Game Day

Production Change - Criticality 2 C2

Change Summary

This production issue is to be used for Gamedays as well as recovery in case of a zonal outage. It outlines the steps to be followed when testing traffic shifts due to zonal outages. Hopefully corrective actions from testing will help us build new steps to take during a real outage.

Gameday execution roles and details

Role Assignee
Change Technician @astarovoytov
Change Reviewer @mattmi
  • Services Impacted - TBD
  • Time tracking - 90 minutes
  • Downtime Component - 30 minutes

This Gameday change will emulate us-east1-d zone outage.

[For Gamedays only] Preparation Tasks

  1. One week before the gameday make an announcement on slack production_engineering channel. Consider also posting this in the appropriate environment channels, staging or production.

    • Example message:
    Next week on [DATE & TIME] we will be executing a Traffic Routing game day. The process will involve moving traffic away from a single zone in `gstg` to test our disaster recovery capabilities and measure if we are still within our RTO & RPO targets set by the  [DR working group](https://handbook.gitlab.com/handbook/company/working-groups/disaster-recovery/) for GitLab.com.
See <https://gitlab.com/gitlab-com/gl-infra/production/-/issues/17274>

  2. Add an event to the GitLab Production calendar.

  3. Notify the release managers on Slack by mentioning @release-managers and referencing this issue and await their acknowledgment.

  4. Notify the eoc on Slack by mentioning @sre-oncall and referencing this issue and wait for approval by adding the eoc_approved label.

    • Example message:
    @release-managers or @sre-oncall LINK_TO_THIS_CR is scheduled for execution.
We will be diverting traffic away from a single zone in gstg to test our disaster recovery capabilities and measure if we are still within our RTO & RPO targets. Kindly review and approve the CR

  5. Post an FYI link of the slack message to the #test-platform channel on slack.

Detailed steps for the change

Change Steps - steps to take to execute the change

Execution

  1. If you are conducting a practice (Gameday) run of this, consider starting a recording of the process now.

  2. Note the start time in UTC in a comment to record this process duration.

  3. Set label changein-progress /label ~change::in-progress

  4. Reconfigure the regional cluster to exclude the affected zone by setting regional_cluster_zones in Terraform to a list of zones that are not impacted

    ❗❗NOTE❗❗ _This takes a while to complete (approximately 30 minutes)

  5. After the above MRs merged and pipelines completed, trigger the chef client run on HAProxy nodes:

    • knife ssh 'name:haproxy* AND chef_environment:gstg' 'sudo chef-client'

  6. Remove the HAProxy instances from the GCP load balancers (this must be done AFTER the above terraform change is applied):

    • Check out the chef-repo repository at: git@gitlab.com:gitlab-com/gl-infra/chef-repo.git
    cd chef-repo
./bin/manage-gcp-lb-haproxy
    • The script will prompt for an environment, then a zone. Select the values correlating to the zone we are removing traffic from.

  7. Disable the HAproxy servers:

    cd chef-repo
./bin/disable-server gstg us-east1-d
    • Validate that all servers in the affected zone have their state set to MAINT. You can use this query to confirm that there are zero backends in the UP state for the zone that you are evacuating.

  8. Note the conclusion time in UTC in a comment to record this process duration.

Validation

Once traffic is restricted to our remaining two zones, let's identify the impact and look for problems.

Wrapping up

  • Re-enable the zonal GKE backend cluster in HAProxy

    cd chef-repo
./bin/enable-server gstg us-east1-d
    • Validate that all servers in the affected zone have their state set to UP. You can use this query to confirm that there are zero backends in the MAINT state in the zone that you are returning to service.

  • Open a MR to revert the change to disable the zone in the regional cluster. (This will also revert the changes made to the GCP loadbalancers with gcloud commands)

  • Set label changecomplete /label ~change::complete

  • Notify the @release-managers and @sre-oncall that the exercise is complete.

  • Compile the real time measurement of this process and update the Recovery Measrements Runbook.

Cleanup

Revert the changes for this Gameday:

Rollback

Rollback steps - steps to be taken in the event of a need to rollback this change

It is estimated that this will take 5m to complete

  • Re-enable HAProxy with Terraform

  • Re-enable HAProxy

    cd chef-repo
./bin/enable-server gstg us-east1-d

  • Set label changecomplete /label ~change::aborted

  • Notify the @release-managers and @sre-oncall that the exercise has been aborted.

Change Reviewer checklist

C4 C3 C2 C1:

  • Check if the following applies:
    • The scheduled day and time of execution of the change is appropriate.
    • The change plan is technically accurate.
    • The change plan includes estimated timing values based on previous testing.
    • The change plan includes a viable rollback plan.
    • The specified metrics/monitoring dashboards provide sufficient visibility for the change.

C2 C1:

  • Check if the following applies:
    • The complexity of the plan is appropriate for the corresponding risk of the change. (i.e. the plan contains clear details).
    • The change plan includes success measures for all steps/milestones during the execution.
    • The change adequately minimizes risk within the environment/service.
    • The performance implications of executing the change are well-understood and documented.
    • The specified metrics/monitoring dashboards provide sufficient visibility for the change.
      • If not, is it possible (or necessary) to make changes to observability platforms for added visibility?
    • The change has a primary and secondary SRE with knowledge of the details available during the change window.
    • The change window has been agreed upon with Release Managers in advance of the change. If the change is planned for APAC hours, this issue has an agreed pre-change approval.
    • The labels blocks deployments and/or blocks feature-flags are applied as necessary.

Change Technician checklist

  • Check if all items below are complete:
    • The change plan is technically accurate.
    • This Change Issue is linked to the appropriate Issue and/or Epic
    • Change has been tested in staging and results are noted in a comment on this issue.
    • A dry-run has been conducted and results are noted in a comment on this issue.
    • The change execution window respects the Production Change Lock periods.
    • For C1 and C2 change issues, the change event is added to the GitLab Production calendar.
    • For C1 and C2 change issues, the SRE on-call has been informed before the change is rolled out. (In the #production channel, mention @sre-oncall and this issue and await their acknowledgement.)
    • For C1 and C2 change issues, the SRE on-call provided approval with the eoc_approved label on the issue.
    • For C1 and C2 change issues, the Infrastructure Manager provided approval with the manager_approved label on the issue.
    • Release managers have been informed prior to any C1, C2, or blocks deployments change being rolled out. (In the #production channel, mention @release-managers and this issue and await their acknowledgement.)
    • There are currently no active incidents that are severity1 or severity2
    • If the change involves doing maintenance on a database host, an appropriate silence targeting the host(s) should be added for the duration of the change.
Edited by Anton Starovoytov