Weekly Reliability (SRE) Team Newsletter – On-call Period: 2022-08-30 - 2022-09-06
<!-- This issue was automatically generated by https://gitlab.com/gitlab-com/gl-infra/oncall-robot-assistant. -->
<!-- Announcements common to all the Reliability (SRE) Teams should be placed in this section. -->
# Announcements
#### [Engineering Week in Review](https://docs.google.com/document/d/1JBdCl3MAOSdlgq3kzzRmtzTsFWsTIQ9iQg0RHhMht6E/edit) Highlights:
<!-- Announcements for each individual SRE Team should be made in their respective sections below. -->
# Team Updates
<!-- xxYZzXcV -->
---
# On-Call During This Period
| Schedule | Username |
| -------- | -------- |
| SRE 8-hour Americas | Nels Nelson |
| SRE 8-hour Americas | Igor Wiedler |
| SRE 8-hour Americas | Marcel Chacon |
| SRE 8-hour APAC | Craig Barrett |
| SRE 8-hour EMEA | Rehab Hassanein |
| SRE 8-hour EMEA | Igor Wiedler |
## PagerDuty Incidents
[See the 1 week report for acknowledged PD pages](https://nonprod-log.gitlab.net/app/dashboards#/view/dacc1d40-1c64-11ec-b8fd-b5d052b1f8cb?_g=(time:(from:'2022-08-30T01:00:00Z',to:'2022-09-06T01:00:00Z'),filters:!((query:(match_phrase:(type.keyword:pagerduty))),(query:(match_phrase:(status.keyword:triggered)))))) ([long-term trend](https://nonprod-log.gitlab.net/goto/a436702d57864666cd0c8867cfaf73e9))
### Alerts Volume
* [Weekly Trend](https://nonprod-log.gitlab.net/goto/4b20cce0-cb91-11ec-b3a6-472d0398dd6e)
* [Monthly Trend](https://nonprod-log.gitlab.net/goto/89e93020-cb91-11ec-b3a6-472d0398dd6e)
* [90 Days Trend by Service](https://nonprod-log.gitlab.net/goto/ed862250-cb91-11ec-b3a6-472d0398dd6e)
### 7 Day Issue Stats
* Oncall issues : **0**
* Access Request : **0**
* Change Issues : **7**
* Incident Issues : **16**
* CorrectiveAction Issues : **1**
#### Change Issues
* 2022-09-02T23:39:02Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7685+
* 2022-09-02T21:58:53Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7684+
* 2022-09-02T09:37:55Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7681+
* 2022-09-02T08:35:52Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7680+
* 2022-09-01T18:17:06Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7676+
* 2022-09-01T11:17:47Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7674+
* 2022-09-01T04:29:22Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7673+
#### Incident Issues
* 2022-09-03T16:53:15Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7688+ | reliability~3760142 | ~"Service::Web" | `https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7688`
* 2022-09-03T09:56:33Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7687+ | reliability~3760141 | ~"Service::Praefect" | `https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7687`
* 2022-09-03T09:15:40Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7686+ | reliability~3760141 | ~"Service::CustomersDot" | `https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7686`
* 2022-09-02T13:39:37Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7683+ | reliability~3760142 | ~"Service::Praefect" | `https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7683`
* 2022-09-02T12:24:53Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7682+ | reliability~3760142 | ~"Service::Praefect" | `https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7682`
* 2022-09-02T08:15:59Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7679+ | reliability~3760140 | ~"Service::Infrastructure" | `https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7679`
* 2022-09-01T19:21:10Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7677+ | reliability~3760142 | ~"Service::Monitoring-Other" | `https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7677`
* 2022-09-01T14:28:11Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7675+ | reliability~3760142 | ~"Service::PVS" | `https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7675`
* 2022-08-31T19:23:22Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7672+ | reliability~3760141 | ~"Service::Woodhouse" | `https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7672`
* 2022-08-31T14:33:05Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7671+ | reliability~3760141 | ~"Service::Gitaly" | `https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7671`
* 2022-08-31T13:44:34Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7670+ | reliability~3760140 | ~"Service::CustomersDot" | `https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7670`
* 2022-08-31T08:01:44Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7669+ | reliability~3760141 | ~"Service::Logging" | `https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7669`
* 2022-08-30T18:12:54Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7667+ | reliability~3760141 | ~"Service::Prometheus" | `https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7667`
* 2022-08-30T12:07:24Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7665+ | reliability~3760140 | ~"Service::Design" | `https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7665`
* 2022-08-30T06:49:19Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7664+ | reliability~3760142 | ~"Service::Frontend" | `https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7664`
* 2022-08-29T07:43:12Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7662+ | reliability~3760142 | ~"Service::Gitaly" | `https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7662`
#### CorrectiveAction Issues
* 2022-08-31T07:54:51Z - https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/16269+
* 2022-08-30T22:31:45Z - https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/16261+
* 2022-08-30T14:05:55Z - https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/16258+
* 2022-08-29T10:51:00Z - https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/16254+
### Open Issue Stats
* [Oncall issues](https://gitlab.com/gitlab-com/infrastructure/issues?scope=all&utf8=%E2%9C%93&state=opened&label_name[]=oncall) : **2**
* [Change issues](https://gitlab.com/gitlab-com/production/issues?scope=all&utf8=%E2%9C%93&state=opened&label_name[]=change) : **28**
* [Incident issues](https://gitlab.com/gitlab-com/production/issues?scope=all&utf8=%E2%9C%93&state=opened&label_name[]=incident) : **9**
* [Access Request](https://gitlab.com/gitlab-com/infrastructure/issues?scope=all&utf8=%E2%9C%93&state=opened&label_name[]=access%20request) : **0**
* [CorrectiveAction](https://gitlab.com/gitlab-com/infrastructure/issues?scope=all&utf8=%E2%9C%93&state=opened&label_name[]=corrective%20action) : **88**
#### Open Change Issues
<details>
<summary>Show/Hide Table</summary>
| Created | Summary |
| ------- | ------- |
| 2022-09-02T23:39:02Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7685+ |
| 2022-09-02T21:58:53Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7684+ |
| 2022-09-02T09:37:55Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7681+ |
| 2022-09-02T08:35:52Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7680+ |
| 2022-09-01T04:29:22Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7673+ |
| 2022-08-26T14:28:22Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7658+ |
| 2022-08-25T14:11:48Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7653+ |
| 2022-08-23T10:33:31Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7636+ |
| 2022-08-19T08:29:12Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7624+ |
| 2022-08-17T19:40:56Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7615+ |
| 2022-08-16T16:43:58Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7607+ |
| 2022-08-16T14:31:39Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7606+ |
| 2022-08-15T10:36:05Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7598+ |
| 2022-08-12T15:04:02Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7586+ |
| 2022-08-12T14:20:37Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7585+ |
| 2022-08-03T14:54:21Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7543+ |
| 2022-08-03T14:00:10Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7542+ |
| 2022-08-01T08:17:13Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7531+ |
| 2022-08-01T08:16:04Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7530+ |
| 2022-08-01T08:15:16Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7529+ |
| 2022-08-01T08:14:38Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7528+ |
| 2022-07-27T14:39:01Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7518+ |
| 2022-07-25T16:23:58Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7506+ |
| 2022-07-22T14:23:23Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7497+ |
| 2022-06-28T16:43:38Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7343+ |
| 2022-06-27T14:16:42Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7334+ |
| 2022-06-06T15:55:01Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7206+ |
| 2022-05-23T16:22:23Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7117+ |
</details>
#### Open Incident Issues
<details>
<summary>Show/Hide Table</summary>
| Created | Summary |
| ------- | ------- |
| 2022-08-31T08:01:44Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7669+ |
| 2022-08-15T16:40:33Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7601+ |
| 2022-08-09T17:31:57Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7570+ |
</details>
#### Open Oncall Issues
<details>
<summary>Show/Hide Table</summary>
| Created | Summary |
| ------- | ------- |
| 2021-09-17T19:35:34Z | https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/14205+ |
| 2020-12-18T22:29:14Z | https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/12200+ |
</details>
#### Issues for Review during Incident Review Meeting
<details>
If there are any incidents you think would be good to review, please add them to the [Agenda](https://docs.google.com/document/d/1Llm9tXHC2dNt_eercRUUXlUyWmOVw00wmXWQQbWvv2c/edit?usp=sharing) for the next meeting.
</details>
issue