# Production Change - Criticality 2 ~C2 ## Change Summary This production issue is to be used for Gamedays as well as recovery in case of a zonal outage. It outlines the steps to be followed when restoring Gitaly VMs in a single zone. This production issue is divided into two sections , one for [planned gamedays](#planned-game-day-gstg) and one for in the event of an [actual outage](#production-outage) ### Execution roles and details <table> <tr> <th>Role</th> <th>Assignee</th> </tr> <tr> <td>Change Technician</td> <td> @roshs <!--woodhouse: '`@{{ .Username }}`'--> </td> </tr> <tr> <td>Change Reviewer</td> <td> @romongbale <!--woodhouse: '@{{ .Reviewer }}'--> </td> </tr> </table> ## Change Details 1. **Services Impacted** - ~&quot;Service::Gitaly&quot; 2. **Time tracking** - 90 minutes 3. **Downtime Component** - <!--woodhouse: '{{ .Downtime }}'--> 30 minutes **Scheduled Date and Time (UTC in format YYYY-MM-DD HH:MM)** - 2026-07-14 16:00 A gameday that involves moving traffic away from a single zone in gstg, and moving Gitaly nodes to a new zone. ## Planned Game Day (GSTG) ## Preparation Tasks ### The Week Prior 1. [x] One week before the gameday make an announcement on slack [#f_gamedays](https://gitlab.enterprise.slack.com/archives/C07PV3F6J1W), copy the link to the [#s_production_engineering](https://gitlab.enterprise.slack.com/archives/C07U6SAKS4D) channel and request approval from `@release-managers`. Consider also sharing this post in the appropriate environment channels; [staging](https://gitlab.enterprise.slack.com/archives/CLM200USV) or [production](https://gitlab.enterprise.slack.com/archives/C101F3796). ```code Next week on [DATE & TIME] we will be executing a Gitaly gameday. The process will involve moving traffic away from a single zone in gstg, and moving Gitaly nodes to a new zone. This should take approximately 90 minutes, the aim for this exercise is to test our disaster recovery capabilities and measure if we are still within our RTO & RPO targets set by the [DR working group](https://handbook.gitlab.com/handbook/company/working-groups/disaster-recovery/) for GitLab.com. See https://gitlab.com/gitlab-com/gl-infra/production/-/issues/17274 ``` 2. [x] Request approval from the Infrastructure manager, wait for approval and confirm by the ~manager_approved label. 3. [x] Make sure an event has been created in [GitLab Production](https://calendar.google.com/calendar/embed?src=gitlab.com_si2ach70eb1j65cnu040m3alq0%40group.calendar.google.com) calendar by setting the Scheduled Date and Time (top section of this issue) and adding the ~change::scheduled label. 4. [x] Ensure you have [PAT](https://staging.gitlab.com/-/user_settings/personal_access_tokens), you would require admin API access , please select `api` , `read_api` and `admin_mode scopes`, also validate if you have [SSH key](https://staging.gitlab.com/-/user_settings/ssh_keys) setup for staging. 5. [x] Confirm that you can run `knife` from your `chef-repo`. You can use to verify the setup ```bash knife status ``` ### Day Of 1. [x] Notify the release managers on _Slack_ by mentioning `@release-managers` and referencing this issue and await their acknowledgment. 2. [x] Notify the eoc on _Slack_ by mentioning `@sre-oncall` and referencing this issue and wait for approval by adding the ~eoc_approved label. - **Example message:** ```code @release-managers or @sre-oncall LINK_TO_THIS_CR is scheduled for execution. We will be taking Gitaly nodes offline in a single zone in staging `gstg` for approximately 30 minutes so as to avoid dataloss. This will only affect projects that are on those Gitaly VMs, it won't be used for new projects. Kindly review and approve the CR. ``` 3. [x] Share this message in the [#staging](https://gitlab.enterprise.slack.com/archives/CLM200USV) channel on Slack. ## Detailed steps for the change ### Change Steps #### Execution 1. [x] Consider starting a recording of the process now. 2. [x] Note the start time in UTC in a comment to record this process duration. 3. [x] Set label ~&quot;change::in-progress&quot; `/label ~change::in-progress` 4. [x] Identify the Gitaly server to run the checksum command - `gitaly-02b-stor-gstg.c.gitlab-gitaly-gstg-164c.internal` is likely to be the storage name for a host, you can cross-reference the [rails configuration](https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/blob/ad3b9819198bc0ce683ebbb4f7831b89871d53b1/releases/gitlab/values/gstg.yaml.gotmpl?page=2#L1466) for validation , additionally assuming we know which zone is affected , we can navigate to [Repository](https://staging.gitlab.com/admin/application_settings/repository) to confirm. 5. [x] Get a list of Gitaly projects by running the following ```bash gcloud projects list --filter="name:gitlab-gitaly-gstg*" ``` 6. [x] Note down the `md5sum` of a repository on the affected storage in a comment on this issue. - `glsh gitaly repositories checksum -s gitaly-02b-stor-gstg.c.gitlab-gitaly-gstg-380a.internal -e gstg -g lots-of-small-projects` - replace `gitaly-02b-stor-gstg.c.gitlab-gitaly-gstg-380a.internal` with the _storage name_ (not necessarily fqdn) of one of the impacted storage. - Note that there are multiple Gitaly projects in `gstg` and `gprd` , in the above example we are calculating the `md5sum` of the _one of the impacted storage_ ie: from the `gitlab-gitaly-gstg-380a` project. - Note that many projects on staging are empty projects, so verify that the checksummed project has some content in it. If the command above returned an empty project, change the `-g` value, which is _not a group name_ but a search value suffixed with a slash (i.e. for the command above it will search for "lots-of-small-projects/"). 7. [x] Use this script to create and push commits to a new branch in [`config-mgmt`](https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt) , [`k8s-workloads/gitlab-com`](https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-com) and [`chef-repo`](https://gitlab.com/gitlab-com/gl-infra/chef-repo). This script creates new commits with the necessary changes in the `tmp` directory. ```bash gcloud auth application-default login --project gitlab-staging-1 cd runbooks/scripts/disaster-recovery bundle bundle exec ruby gitaly-replace-nodes.rb -e gstg -z us-east1-b --app-config --commit --push # Where all of the Gitaly nodes in us-east1-b will be redistributed among the remaining zones. ``` - You can use the `-w` parameter to pass a path where the repositories with be cloned. ```bash gcloud auth application-default login --project gitlab-staging-1 cd runbooks/scripts/disaster-recovery bundle bundle exec ruby gitaly-replace-nodes.rb -e gstg -z us-east1-b --app-config --commit --push -w temp_dir # Where all of the Gitaly nodes in us-east1-b will be redistributed among the remaining zones. ``` 8. [x] Create merge requests in [`config-mgmt`](https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt) , [`k8s-workloads/gitlab-com`](https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-com) and [`chef-repo`](https://gitlab.com/gitlab-com/gl-infra/chef-repo) against the new branches created by the script, for example. Since the commits have already been pushed, navigating to the repositories will give you an option to create the MR from the UI. - [x] MR that adds Gitaly servers within the available zone in config-mgmt. - Example `gstg`: https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/merge_requests/8788/diffs - Example `gprd`: https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/merge_requests/8631/diffs <details> <summary> **:exclamation::exclamation:IMPORTANT NOTE: :exclamation::exclamation:** </summary> When testing the ability to restore production instances, this should be the only MR that gets created and merged. We cannot perform an application switch over to the new nodes during a game day in GPRD, therefore, stop at this step. _In the above :arrow_up: example MR, only one set of nodes were provisioned, in an actual recovery scenario, there will be 3 total node blocks that need to be created._ </details> - [x] MR to update application configuration in `k8s-workloads/gitlab-com` - Example: https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/merge_requests/3720/diffs - [x] MR to update application configuration in `chef-repo` - Example: https://gitlab.com/gitlab-com/gl-infra/chef-repo/-/merge_requests/4889/diffs <!--ADD MR LINKS HERE--> 9. [x] Record the current weights of the affected storage in a comment in this issue. The lists of storage can be found from the MR in the step above - With an admin account, navigate to [Repository Storage Settings](https://staging.gitlab.com/admin/application_settings/repository) and record the weights of the affected storages that correspond to the instances being replaced. 10. [x] Set the weights to zero on the affected storages. - With an admin account, navigate to [Repository Storage Settings](https://staging.gitlab.com/admin/application_settings/repository) and set the weight to 0 for the affected storages that correspond to the instances being replaced. 11. [x] Stop the Gitaly nodes and create new snapshots. (Since we currently have two Gitaly projects in `gstg`) :exclamation::exclamation: **NOTE:** :exclamation::exclamation: The zone, instance and disk names in these commands are examples. They need to be modified to match the values corresponding to the instances you are replacing. Ensure you remove the place holders `<>` when you update the correct names. We are only taking snapshots for the data disk as that's where the customer data lives. The other disks are ephemeral and are managed by Chef and can easily be recreated. ```bash gcloud compute instances stop --project=gitlab-gitaly-gstg-380a --zone="us-east1-b" "gitaly-02b-stor-gstg" gcloud compute snapshots create "file-gitaly-02-gameday-snapshot" --source-disk="gitaly-02b-stor-gstg-data" --description="Part of https://gitlab.com/gitlab-com/gl-infra/production/-/issues/22417" --project=gitlab-gitaly-gstg-380a --source-disk-zone="us-east1-b" gcloud compute instances stop --project=gitlab-gitaly-gstg-164c --zone="us-east1-b" "gitaly-02b-stor-gstg" gcloud compute snapshots create "file-gitaly-02-gameday-snapshot" --source-disk="gitaly-02b-stor-gstg-data" --description="Part of https://gitlab.com/gitlab-com/gl-infra/production/-/issues/22417" --project=gitlab-gitaly-gstg-164c --source-disk-zone="us-east1-b" ``` 12. [x] Verify that the repositories on the storage are no longer available and paste the output as a comment. The same `glsh gitaly repositories checksum` command from above should exit with an error similar to this one. ```bash glsh gitaly repositories checksum -s gitaly-02-stor-gstg.c.gitlab-gitaly-gstg-164c.internal -e gstg -g gitlab-com ✘ curl exited non-zero! curl: (22) The requested URL returned error: 500 {"message":"500 Internal Server Error"} ``` 13. [x] Check and validate that the terraform plan `config-mgmt` MRs Atlantis run is creating disks using the snapshots that were just taken in the previous step. - [x] Rerun the terraform plan and verify the snapshots are correct - [x] Review the plan and identify the source snapshot for the Data disks and that the snapshots are the snapshots created earlier in this process. - [x] :exclamation::exclamation: **NOTE:** :exclamation::exclamation: There might be scheduled snapshots taken after the manually created snapshots which will probably be **0B** in size and might be in the Terraform plan instead of our manually created snapshot. - [x] Merge the \[config-mgmt MR\]\[config-mgmt\] to provision the new Gitaly instances if things appear correct. - [x] :exclamation::exclamation: **NOTE:** :exclamation::exclamation: _This step above :arrow_up: will take 10-20 minutes to complete_ #### Validation Gitaly VMs 1. [x] Wait for the instances to be built and Chef to converge. - [x] [Confirm that chef runs have completed](https://dashboards.gitlab.net/explore?schemaVersion=1&panes=%7B%229oq%22:%7B%22datasource%22:%22e58c2f51-20f8-4f4b-ad48-2968782ca7d6%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22max%28chef_client_last_run_timestamp_seconds%7Benv%3D%5C%22gstg%5C%22,%20type%3D~%5C%22gitaly%5C%22%7D%29%20by%20%28fqdn%29%22,%22range%22:true,%22instant%22:true,%22datasource%22:%7B%22type%22:%22prometheus%22,%22uid%22:%22e58c2f51-20f8-4f4b-ad48-2968782ca7d6%22%7D,%22editorMode%22:%22code%22,%22legendFormat%22:%22__auto%22%7D%5D,%22range%22:%7B%22from%22:%22now-1h%22,%22to%22:%22now%22%7D%7D%7D&orgId=1) for new storages, it can take up to 20 minutes before they show up. - [x] [Confirm the VMs have been successfully run chef](https://dashboards.gitlab.net/goto/yC_OaxmHR?orgId=1) - [x] Validate if Bootstrapping is completed - We could also tail bootstrap the _latest logs_ example: `tail -f /var/tmp/bootstrap-20231108-133642.log`,`20231108` represents the date of creation of the log file in `yyyy-mm-dd` format, `133642` represents time in UTC in `HH-MM-SS`format. - Bootstrapping is completed once the machine has rebooted, and the most recent bootstrap script log from the game day has reported: `Bootstrap finished` <details> <summary>Trouble shooting tips</summary> - Tail the serial output to confirm that the start-up script executed successfully. ```bash gcloud compute --project=$project instances tail-serial-port-output $instance_name --zone=$zone --port=1` ``` the variables `$project` represents the gitaly project e.g `gitaly-gstg-380a` for the gitaly storages, `$instance_name` represent the instance e.g `gitaly-01a-stor-gstg`, and `$zone` represents the recovery zone e.g `us-east1-c`. </details> 2. [x] ssh into the Gitaly VMs: - Ensure a separate disk has been mounted to `/var/opt/gitlab` - shell: `knife ssh --no-host-key-verify -C 10 "role:gstg-base-stor-gitaly" "mount | grep /opt/gitlab"` - shell: `mount | grep /opt/gitlab` - [Mimir](https://dashboards.gitlab.net/explore?schemaVersion=1&panes=%7B%22u11%22:%7B%22datasource%22:%22e58c2f51-20f8-4f4b-ad48-2968782ca7d6%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22node_filesystem_free_bytes%7Benv%3D%5C%22gstg%5C%22,%20type%3D%5C%22gitaly%5C%22,%20mountpoint%3D%5C%22%2Fvar%2Fopt%2Fgitlab%5C%22,%20fqdn%3D~%5C%22gitaly-02a-stor-gstg.%2A%5C%22%7D%20%2F%20node_filesystem_size_bytes%7Benv%3D%5C%22gstg%5C%22,%20type%3D%5C%22gitaly%5C%22,%20mountpoint%3D%5C%22%2Fvar%2Fopt%2Fgitlab%5C%22,%20fqdn%3D~%5C%22gitaly-02a-stor-gstg.%2A%5C%22%7D%22,%22range%22:true,%22instant%22:true,%22datasource%22:%7B%22type%22:%22prometheus%22,%22uid%22:%22e58c2f51-20f8-4f4b-ad48-2968782ca7d6%22%7D,%22editorMode%22:%22code%22,%22legendFormat%22:%22__auto%22%7D%5D,%22range%22:%7B%22from%22:%22now-1h%22,%22to%22:%22now%22%7D%7D%7D&orgId=1) :exclamation::exclamation:**NOTE**:exclamation::exclamation: _The graph above is an example to offer guidance you may be required to change some parameters e.g. `fqdn`._ - Execute `sudo gitlab-ctl status` to validate that the servers are up ## **POINT OF NO RETURN** ### Add the new Storage to the Rails and Gitaly configuration 1. [x] Merge the `k8s-workload/gitlab-com` \[MR\]\[gitlab-com\] 2. [x] Merge the `chef-repo` \[MR\]\[chef-repo\] 3. [x] Once the `chef-repo` pipeline succeeds, force a `chef-client` run across the entire Gitaly fleet. ```bash knife ssh -t 3 -C 10 'chef_environment:gstg AND (name:gitaly* OR name:file-hdd*)' 'sudo chef-client' ``` #### Validate that the new nodes are receiving traffic 1. [x] https://dashboards.gitlab.net/d/gitaly-host-detail/gitaly3a-host-detail?orgId=1&var-PROMETHEUS_DS=mimir-gitlab-gstg&var-environment=gstg&var-fqdn=gitaly-02c-stor-gstg.c.gitlab-gitaly-gstg-380a.internal&viewPanel=1687078843 2. [x] https://dashboards.gitlab.net/d/gitaly-host-detail/gitaly3a-host-detail?orgId=1&var-PROMETHEUS_DS=mimir-gitlab-gstg&var-environment=gstg&var-fqdn=gitaly-02c-stor-gstg.c.gitlab-gitaly-gstg-164c.internal&viewPanel=1687078843 :exclamation::exclamation:**NOTE**:exclamation::exclamation: _The dashboards above are examples to offer guidance you may be required to change some parameters e.g. `fqdn`._ 3. [x] Validate a project can be created on the new Storage. - [x] `glsh gitaly storages validate -e gstg gitaly-02c-stor-gstg.c.gitlab-gitaly-gstg-164c.internal` - [x] `glsh gitaly storages validate -e gstg gitaly-02c-stor-gstg.c.gitlab-gitaly-gstg-380a.internal` :exclamation::exclamation:**NOTE**:exclamation::exclamation: _Please note it may take several minutes for validation commands to give reliable output._ 4. [x] Validate that the repository we previously collected an `md5sum` from is available, and the checksums match. Copy the md5sum line as a comment in this issue. - `glsh gitaly repositories checksum -s gitaly-02-stor-gstg.c.gitlab-gitaly-gstg-164c.internal -e gstg -g gitlab-com` :exclamation::exclamation:**NOTE**:exclamation::exclamation: _Remember to replace the storage name with the one that is having hosts migrated for. The values used above are examples to offer guidance._ 5. [x] Validate that the [number of errors](https://dashboards.gitlab.net/d/general-triage/general3a-platform-triage?var-PROMETHEUS_DS=e58c2f51-20f8-4f4b-ad48-2968782ca7d6&var-environment=gstg&var-stage=main&from=now-1h&to=now&viewPanel=3) returned by the application are at a nominal level. 6. [x] Restore the weights of the affected storages. 7. [x] Note the conclusion time in UTC in a comment to record this process duration. 8. [x] Collect timing information - [x] Collect bootstrap timing information - shell: `for node in $(knife node list | grep -P 'gitaly-0\da-stor-gstg'); do path/to/runbooks/scripts/find-bootstrap-duration.sh $node ; done` - Note in a comment the results - [x] Collect apply and completion times for the Terraform apply - Note in a comment the times when the apply was triggered and when it was posted that the apply was completed #### Clean up - [x] Create and merge the MR to the config-mgmt repo, replacing the now unused Gitaly instances with our newly created Gitaly nodes in `gitaly-multi-project.tf`. If the Gitaly node replacement ruby script was used, ensure `gitaly-recovery-nodes.tf` has the original contents. This needs to be done in two steps: - [ ] Remove deletion protection from the existing nodes. - Example MR :point_right: https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/merge_requests/9251 - Ensure that the `data_disk_snapshot_search_string` attribute is also removed from the newly created instances. - [ ] Second MR to remove the VMs - Example MR: https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/merge_requests/9252 :exclamation::exclamation:**NOTE**:exclamation::exclamation: _When deleting Gitaly instances, you need to add the \~approve-policies label and comment `atlantis approve_policies` in the MR to bypass the policies before applying with `atlantis apply`._ - [x] Remove any data disk snapshots that were taken prior to restoring the nodes. These will not be removed automatically. - Verify details about the snapshots ```bash gcloud compute snapshots describe [SNAPSHOT_NAME] ``` - Delete the snapshots ```bash gcloud compute snapshots delete [SNAPSHOT_NAME] ``` :exclamation::exclamation: **Warning:** :exclamation::exclamation: Deleting a snapshot is irreversible. You can't recover a deleted snapshot. #### Wrapping up - [x] Notify the `@release-managers` and `@sre-oncall` that the exercise is complete. :exclamation::exclamation:**NOTE**:exclamation::exclamation: _Ensure all unused Gitaly nodes have been deleted before signalling completion._ - [x] Set label ~&quot;change::complete&quot; `/label ~change::complete` - [x] Compile the real-time measurement of this process and update the [Recovery Measurements Runbook](https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/disaster-recovery/recovery-measurements.md?ref_type=heads). ## Rollback ### Rollback steps - steps to be taken in the event of a need to rollback this change If the **point of no return has not been reached**, and no impactful changes have been applied: - [ ] Notify `@release-managers` and `@sre-oncall` that the exercise is being aborted. - [ ] Set label ~&quot;change::aborted&quot; using `/label ~change::aborted`. - [ ] Close the MRs without merging them - [ ] Start the VMs again - [ ] Restore storage weights back to their previous values - [ ] Document the reason for aborting in the issue, including any observations or anomalies. - [ ] No further action is required — the change is safely aborted. If the **point of no return _has_ been reached**, and changes have already been applied: - [ ] Notify `@release-managers` and `@sre-oncall` that the rollback process is starting. - [ ] Set label ~&quot;change::aborted&quot; using `/label ~change::aborted`. - [ ] Follow the cleanup and recovery steps outlined above to revert or stabilize the environment. - [ ] Document the reason for rollback in the issue, and provide relevant logs or output. ## Monitoring ### Key metrics to observe - Completed Chef runs: [Staging](https://dashboards.gitlab.net/explore?schemaVersion=1&panes=%7B%22owy%22:%7B%22datasource%22:%22e58c2f51-20f8-4f4b-ad48-2968782ca7d6%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22max%28chef_client_last_run_timestamp_seconds%7Benv%3D%5C%22gstg%5C%22,%20type%3D%5C%22gitaly%5C%22%7D%29%20by%20%28fqdn%29%22,%22range%22:true,%22instant%22:true,%22datasource%22:%7B%22type%22:%22prometheus%22,%22uid%22:%22e58c2f51-20f8-4f4b-ad48-2968782ca7d6%22%7D,%22editorMode%22:%22code%22,%22legendFormat%22:%22__auto%22%7D%5D,%22range%22:%7B%22from%22:%22now-1h%22,%22to%22:%22now%22%7D%7D%7D&orgId=1) | [Production](https://dashboards.gitlab.net/explore?schemaVersion=1&panes=%7B%22owy%22:%7B%22datasource%22:%22e58c2f51-20f8-4f4b-ad48-2968782ca7d6%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22max%28chef_client_last_run_timestamp_seconds%7Benv%3D%5C%22gprd%5C%22,%20type%3D%5C%22gitaly%5C%22%7D%29%20by%20%28fqdn%29%22,%22range%22:true,%22instant%22:true,%22datasource%22:%7B%22type%22:%22prometheus%22,%22uid%22:%22e58c2f51-20f8-4f4b-ad48-2968782ca7d6%22%7D,%22editorMode%22:%22code%22,%22legendFormat%22:%22__auto%22%7D%5D,%22range%22:%7B%22from%22:%22now-1h%22,%22to%22:%22now%22%7D%7D%7D&orgId=1) - Gitaly Dashboard: [Staging](https://dashboards.gitlab.net/d/gitaly-main/gitaly3a-overview?from=now-6h&to=now&var-PROMETHEUS_DS=e58c2f51-20f8-4f4b-ad48-2968782ca7d6&var-environment=gstg&var-stage=main&orgId=1) | [Production](https://dashboards.gitlab.net/d/gitaly-main/gitaly3a-overview?from=now-6h&to=now&var-PROMETHEUS_DS=e58c2f51-20f8-4f4b-ad48-2968782ca7d6&var-environment=gprd&var-stage=main&orgId=1) - Gitaly RPS by FQDN: [Staging](https://dashboards.gitlab.net/explore?schemaVersion=1&panes=%7B%22owy%22:%7B%22datasource%22:%22e58c2f51-20f8-4f4b-ad48-2968782ca7d6%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22sum%20by%20%28fqdn%29%20%28%5Cn%20%20rate%28gitaly_service_client_requests_total%7Benv%3D%5C%22gstg%5C%22,fqdn%21%3D%5C%22%5C%22,stage%3D%5C%22main%5C%22,type%3D%5C%22gitaly%5C%22%7D%5B1m%5D%29%5Cn%29%5Cn%20%3E%200%22,%22range%22:true,%22instant%22:true,%22datasource%22:%7B%22type%22:%22prometheus%22,%22uid%22:%22e58c2f51-20f8-4f4b-ad48-2968782ca7d6%22%7D,%22editorMode%22:%22code%22,%22legendFormat%22:%22__auto%22%7D%5D,%22range%22:%7B%22from%22:%22now-1h%22,%22to%22:%22now%22%7D%7D%7D&orgId=1) | [Production](https://dashboards.gitlab.net/explore?schemaVersion=1&panes=%7B%22owy%22:%7B%22datasource%22:%22e58c2f51-20f8-4f4b-ad48-2968782ca7d6%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22sum%20by%20%28fqdn%29%20%28%5Cn%20%20rate%28gitaly_service_client_requests_total%7Benv%3D%5C%22gprd%5C%22,fqdn%21%3D%5C%22%5C%22,stage%3D%5C%22main%5C%22,type%3D%5C%22gitaly%5C%22%7D%5B1m%5D%29%5Cn%29%5Cn%20%3E%200%22,%22range%22:true,%22instant%22:true,%22datasource%22:%7B%22type%22:%22prometheus%22,%22uid%22:%22e58c2f51-20f8-4f4b-ad48-2968782ca7d6%22%7D,%22editorMode%22:%22code%22,%22legendFormat%22:%22__auto%22%7D%5D,%22range%22:%7B%22from%22:%22now-1h%22,%22to%22:%22now%22%7D%7D%7D&orgId=1) - Gitaly Errors by FQDN: [Staging](https://dashboards.gitlab.net/explore?schemaVersion=1&panes=%7B%22owy%22:%7B%22datasource%22:%22e58c2f51-20f8-4f4b-ad48-2968782ca7d6%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22sum%20by%20%28fqdn%29%20%28%5Cn%20%20label_replace%28rate%28gitaly_service_client_requests_total%7Benv%3D%5C%22gstg%5C%22,environment%3D%5C%22gstg%5C%22,fqdn%21%3D%5C%22%5C%22,grpc_code%21~%5C%22AlreadyExists%7CCanceled%7CDeadlineExceeded%7CFailedPrecondition%7CInvalidArgument%7CNotFound%7COK%7CPermissionDenied%7CResourceExhausted%7CUnauthenticated%5C%22,stage%3D%5C%22main%5C%22,type%3D%5C%22gitaly%5C%22%7D%5B1m%5D%29,%20%5C%22_c%5C%22,%20%5C%220%5C%22,%20%5C%22%5C%22,%20%5C%22%5C%22%29%5Cn%20%20or%5Cn%20%20label_replace%28rate%28gitaly_service_client_requests_total%7Bdeadline_type%21%3D%5C%22limited%5C%22,env%3D%5C%22gstg%5C%22,environment%3D%5C%22gstg%5C%22,fqdn%21%3D%5C%22%5C%22,grpc_code%3D%5C%22DeadlineExceeded%5C%22,stage%3D%5C%22main%5C%22,type%3D%5C%22gitaly%5C%22%7D%5B1m%5D%29,%20%5C%22_c%5C%22,%20%5C%221%5C%22,%20%5C%22%5C%22,%20%5C%22%5C%22%29%5Cn%29%5Cn%20%3E%200%22,%22range%22:true,%22instant%22:true,%22datasource%22:%7B%22type%22:%22prometheus%22,%22uid%22:%22e58c2f51-20f8-4f4b-ad48-2968782ca7d6%22%7D,%22editorMode%22:%22code%22,%22legendFormat%22:%22__auto%22%7D%5D,%22range%22:%7B%22from%22:%22now-1h%22,%22to%22:%22now%22%7D%7D%7D&orgId=1) | [Production](https://dashboards.gitlab.net/explore?schemaVersion=1&panes=%7B%22owy%22:%7B%22datasource%22:%22e58c2f51-20f8-4f4b-ad48-2968782ca7d6%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22sum%20by%20%28fqdn%29%20%28%5Cn%20%20label_replace%28rate%28gitaly_service_client_requests_total%7Benv%3D%5C%22gprd%5C%22,fqdn%21%3D%5C%22%5C%22,grpc_code%21~%5C%22AlreadyExists%7CCanceled%7CDeadlineExceeded%7CFailedPrecondition%7CInvalidArgument%7CNotFound%7COK%7CPermissionDenied%7CResourceExhausted%7CUnauthenticated%5C%22,stage%3D%5C%22main%5C%22,type%3D%5C%22gitaly%5C%22%7D%5B1m%5D%29,%20%5C%22_c%5C%22,%20%5C%220%5C%22,%20%5C%22%5C%22,%20%5C%22%5C%22%29%5Cn%20%20or%5Cn%20%20label_replace%28rate%28gitaly_service_client_requests_total%7Bdeadline_type%21%3D%5C%22limited%5C%22,env%3D%5C%22gprd%5C%22,fqdn%21%3D%5C%22%5C%22,grpc_code%3D%5C%22DeadlineExceeded%5C%22,stage%3D%5C%22main%5C%22,type%3D%5C%22gitaly%5C%22%7D%5B1m%5D%29,%20%5C%22_c%5C%22,%20%5C%221%5C%22,%20%5C%22%5C%22,%20%5C%22%5C%22%29%5Cn%29%5Cn%20%3E%200%22,%22range%22:true,%22instant%22:true,%22datasource%22:%7B%22type%22:%22prometheus%22,%22uid%22:%22e58c2f51-20f8-4f4b-ad48-2968782ca7d6%22%7D,%22editorMode%22:%22code%22,%22legendFormat%22:%22__auto%22%7D%5D,%22range%22:%7B%22from%22:%22now-1h%22,%22to%22:%22now%22%7D%7D%7D&orgId=1) ## Change Reviewer checklist <!--To be filled out by the reviewer.--> ~C4 ~C3 ~C2 ~C1: - [ ] Check if the following applies: - The **scheduled day and time** of execution of the change is appropriate. - The [change plan](#detailed-steps-for-the-change) is technically accurate. - The change plan includes **estimated timing values** based on previous testing. - The change plan includes a viable [rollback plan](#rollback). - The specified [metrics/monitoring dashboards](#key-metrics-to-observe) provide sufficient visibility for the change. ~C2 ~C1: - [ ] Check if the following applies: - The complexity of the plan is appropriate for the corresponding risk of the change. (i.e. the plan contains clear details). - The change plan includes success measures for all steps/milestones during the execution. - The change adequately minimizes risk within the environment/service. - The performance implications of executing the change are well-understood and documented. - The specified metrics/monitoring dashboards provide sufficient visibility for the change. - If not, is it possible (or necessary) to make changes to observability platforms for added visibility? - The change has a primary and secondary SRE with knowledge of the details available during the change window. - The change window has been agreed upon with Release Managers in advance of the change. If the change is planned for APAC hours, this issue has an agreed pre-change approval. - The labels ~&quot;blocks deployments&quot; and/or ~&quot;blocks feature-flags&quot; are applied as necessary. ## Change Technician checklist <!--To find out who is on-call, use the `@sre-oncall` handle in slack--> - [x] Check if all items below are complete: - The [change plan](#detailed-steps-for-the-change) is technically accurate. - This Change Issue is linked to the appropriate Issue and/or Epic - Change has been tested in staging and results are noted in a comment on this issue. - A dry-run has been conducted and results are noted in a comment on this issue. - The change execution window respects the [Production Change Lock periods](https://about.gitlab.com/handbook/engineering/infrastructure/change-management/#production-change-lock-pcl). - For ~C1 and ~C2 change issues, the change event is added to the [GitLab Production](https://calendar.google.com/calendar/embed?src=gitlab.com_si2ach70eb1j65cnu040m3alq0%40group.calendar.google.com) calendar. - For ~C1 and ~C2 change issues, the SRE on-call has been informed before the change is rolled out. (In the #production channel, mention `@sre-oncall` and this issue and await their acknowledgement.) - For ~C1 and ~C2 change issues, the SRE on-call provided approval with the ~eoc_approved label on the issue. - For ~C1 and ~C2 change issues, the Infrastructure Manager provided approval with the ~manager_approved label on the issue. - Release managers have been informed prior to any C1, C2, or ~&quot;blocks deployments&quot; change being rolled out. (In the #production channel, mention `@release-managers` and this issue and await their acknowledgement.) - There are currently no [active incidents](https://gitlab.com/gitlab-com/gl-infra/production/-/issues?scope=all&utf8=%E2%9C%93&state=opened&label_name%5B%5D=Incident%3A%3AActive) that are ~severity::1 or ~severity::2 - If the change involves doing maintenance on a database host, an appropriate silence targeting the host(s) should be added for the duration of the change.