Weekly Reliability (SRE) Team Newsletter – On-call Period: 2023-01-24 - 2023-01-31
<!-- This issue was automatically generated by https://gitlab.com/gitlab-com/gl-infra/oncall-robot-assistant. -->
<!-- Announcements common to all the Reliability (SRE) Teams should be placed in this section. -->
# Announcements
#### [Engineering Week in Review](https://docs.google.com/document/d/1JBdCl3MAOSdlgq3kzzRmtzTsFWsTIQ9iQg0RHhMht6E/edit) Highlights:
<!-- Announcements for each individual SRE Team should be made in their respective sections below. -->
# Team Updates
<!-- xxYZzXcV -->
---
# On-Call During This Period
| Schedule | Username |
| -------- | -------- |
| SRE 8-hour Americas | Cindy Pallares |
| SRE 8-hour Americas | Cameron McFarland |
| SRE 8-hour Americas | Marcel Chacon |
| SRE 8-hour APAC | Devin Sylva |
| SRE 8-hour EMEA | Ahmad Sherif |
| SRE 8-hour EMEA | Steve Azzopardi |
## PagerDuty Incidents
[See the 1 week report for acknowledged PD pages](https://nonprod-log.gitlab.net/app/dashboards#/view/dacc1d40-1c64-11ec-b8fd-b5d052b1f8cb?_g=(time:(from:'2023-01-24T02:00:00Z',to:'2023-01-31T02:00:00Z'),filters:!((query:(match_phrase:(type.keyword:pagerduty))),(query:(match_phrase:(status.keyword:triggered)))))) ([long-term trend](https://nonprod-log.gitlab.net/goto/a436702d57864666cd0c8867cfaf73e9))
### Alerts Volume
* [Weekly Trend](https://nonprod-log.gitlab.net/goto/4b20cce0-cb91-11ec-b3a6-472d0398dd6e)
* [Monthly Trend](https://nonprod-log.gitlab.net/goto/89e93020-cb91-11ec-b3a6-472d0398dd6e)
* [90 Days Trend by Service](https://nonprod-log.gitlab.net/goto/ed862250-cb91-11ec-b3a6-472d0398dd6e)
### 7 Day Issue Stats
* Oncall issues : **0**
* Access Request : **0**
* Change Issues : **16**
* Incident Issues : **19**
* CorrectiveAction Issues : **0**
#### Change Issues
* 2023-01-28T08:03:30Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8312+
* 2023-01-27T19:43:52Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8310+
* 2023-01-27T16:50:11Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8309+
* 2023-01-27T16:35:51Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8308+
* 2023-01-27T14:40:52Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8307+
* 2023-01-27T10:58:07Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8305+
* 2023-01-26T19:19:21Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8303+
* 2023-01-25T14:24:03Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8291+
* 2023-01-25T13:55:33Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8290+
* 2023-01-24T16:42:20Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8286+
* 2023-01-24T07:40:45Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8284+
* 2023-01-24T02:24:10Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8282+
* 2023-01-24T02:15:54Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8281+
* 2023-01-23T20:31:22Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8280+
* 2023-01-23T15:26:26Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8279+
* 2023-01-23T07:52:37Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8276+
#### Incident Issues
* 2023-01-28T18:54:29Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8314+ | reliability~3760142 | ~"Service::Thanos" | `https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8314`
* 2023-01-28T13:57:43Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8313+ | reliability~3760141 | ~"Service::Websockets" | `https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8313`
* 2023-01-27T21:22:17Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8311+ | reliability~3760140 | ~"Service::Patroni" | `https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8311`
* 2023-01-27T13:10:56Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8306+ | reliability~3760142 | ~"Service::Prometheus" | `https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8306`
* 2023-01-26T22:45:52Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8304+ | reliability~3760141 | ~"Service::GitLab Rails" | `https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8304`
* 2023-01-26T17:40:56Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8302+ | reliability~3760141 | ~"Service::Websockets" | `https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8302`
* 2023-01-26T11:07:14Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8301+ | reliability~3760141 | ~"Service::CustomersDot" | `https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8301`
* 2023-01-26T10:57:37Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8300+ | reliability~3760141 | | `https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8300`
* 2023-01-26T10:06:26Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8299+ | reliability~3760141 | ~"Service::Logging" | `https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8299`
* 2023-01-26T09:42:21Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8298+ | reliability~3760141 | ~"Service::Workhorse" | `https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8298`
* 2023-01-26T08:40:08Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8296+ | reliability~3760142 | | `https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8296`
* 2023-01-26T06:55:30Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8295+ | reliability~3760142 | ~"Service::Thanos" | `https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8295`
* 2023-01-25T22:27:27Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8294+ | reliability~3760141 | ~"Service::ExternalDNS" | `https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8294`
* 2023-01-25T21:31:53Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8293+ | reliability~3760141 | ~"Service::Logging" | `https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8293`
* 2023-01-25T17:29:24Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8292+ | reliability~3760141 | | `https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8292`
* 2023-01-25T09:08:15Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8289+ | reliability~3760142 | ~"Service::PVS" | `https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8289`
* 2023-01-24T17:18:04Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8287+ | | | `https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8287`
* 2023-01-24T13:28:30Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8285+ | reliability~3760140 | ~"Service::Web" | `https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8285`
* 2023-01-23T08:50:51Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8277+ | reliability~3760142 | ~"Service::Logging" | `https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8277`
#### CorrectiveAction Issues
* 2023-01-26T18:29:16Z - https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/17184+
* 2023-01-25T07:57:05Z - https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/17171+
* 2023-01-24T04:34:14Z - https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/17164+
### Open Issue Stats
* [Oncall issues](https://gitlab.com/gitlab-com/infrastructure/issues?scope=all&utf8=%E2%9C%93&state=opened&label_name[]=oncall) : **2**
* [Change issues](https://gitlab.com/gitlab-com/production/issues?scope=all&utf8=%E2%9C%93&state=opened&label_name[]=change) : **22**
* [Incident issues](https://gitlab.com/gitlab-com/production/issues?scope=all&utf8=%E2%9C%93&state=opened&label_name[]=incident) : **7**
* [Access Request](https://gitlab.com/gitlab-com/infrastructure/issues?scope=all&utf8=%E2%9C%93&state=opened&label_name[]=access%20request) : **0**
* [CorrectiveAction](https://gitlab.com/gitlab-com/infrastructure/issues?scope=all&utf8=%E2%9C%93&state=opened&label_name[]=corrective%20action) : **91**
#### Open Change Issues
<details>
<summary>Show/Hide Table</summary>
| Created | Summary |
| ------- | ------- |
| 2023-01-27T19:43:52Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8310+ |
| 2023-01-27T16:50:11Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8309+ |
| 2023-01-26T19:19:21Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8303+ |
| 2023-01-25T14:24:03Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8291+ |
| 2023-01-25T13:55:33Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8290+ |
| 2023-01-19T14:51:06Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8263+ |
| 2023-01-10T07:17:05Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8225+ |
| 2022-12-22T03:57:59Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8186+ |
| 2022-12-01T00:14:18Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8113+ |
| 2022-11-14T23:01:34Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8048+ |
| 2022-10-18T07:03:19Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7893+ |
| 2022-10-18T06:45:39Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7892+ |
| 2022-10-07T22:14:21Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7848+ |
| 2022-10-06T07:05:45Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7840+ |
| 2022-10-05T00:00:51Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7834+ |
| 2022-09-02T09:37:55Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7681+ |
| 2022-08-25T14:11:48Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7653+ |
| 2022-08-17T19:40:56Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7615+ |
| 2022-08-16T16:43:58Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7607+ |
| 2022-07-22T14:23:23Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7497+ |
| 2022-06-06T15:55:01Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7206+ |
| 2022-05-23T16:22:23Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7117+ |
</details>
#### Open Incident Issues
<details>
<summary>Show/Hide Table</summary>
| Created | Summary |
| ------- | ------- |
| 2023-01-28T18:54:29Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8314+ |
| 2023-01-27T21:22:17Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8311+ |
| 2023-01-26T08:40:08Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8296+ |
| 2023-01-24T17:18:04Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8287+ |
</details>
#### Open Oncall Issues
<details>
<summary>Show/Hide Table</summary>
| Created | Summary |
| ------- | ------- |
| 2021-09-17T19:35:34Z | https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/14205+ |
| 2020-12-18T22:29:14Z | https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/12200+ |
</details>
#### Issues for Review during Incident Review Meeting
<details>
If there are any incidents you think would be good to review, please add them to the [Agenda](https://docs.google.com/document/d/1Llm9tXHC2dNt_eercRUUXlUyWmOVw00wmXWQQbWvv2c/edit?usp=sharing) for the next meeting.
</details>
issue