Weekly Reliability (SRE) Team Newsletter – On-call Period: 2022-11-08 - 2022-11-15
<!-- This issue was automatically generated by https://gitlab.com/gitlab-com/gl-infra/oncall-robot-assistant. -->
<!-- Announcements common to all the Reliability (SRE) Teams should be placed in this section. -->
# Announcements
#### [Engineering Week in Review](https://docs.google.com/document/d/1JBdCl3MAOSdlgq3kzzRmtzTsFWsTIQ9iQg0RHhMht6E/edit) Highlights:
<!-- Announcements for each individual SRE Team should be made in their respective sections below. -->
# Team Updates
<!-- xxYZzXcV -->
---
# On-Call During This Period
| Schedule | Username |
| -------- | -------- |
| SRE 8-hour Americas | Alex Hanselka |
| SRE 8-hour Americas | Hendrik Meyer |
| SRE 8-hour APAC | Devin Sylva |
| SRE 8-hour EMEA | Alejandro Rodríguez |
| SRE 8-hour EMEA | Igor Wiedler |
| SRE 8-hour EMEA | Steve Azzopardi |
## PagerDuty Incidents
[See the 1 week report for acknowledged PD pages](https://nonprod-log.gitlab.net/app/dashboards#/view/dacc1d40-1c64-11ec-b8fd-b5d052b1f8cb?_g=(time:(from:'2022-11-08T02:00:00Z',to:'2022-11-15T02:00:00Z'),filters:!((query:(match_phrase:(type.keyword:pagerduty))),(query:(match_phrase:(status.keyword:triggered)))))) ([long-term trend](https://nonprod-log.gitlab.net/goto/a436702d57864666cd0c8867cfaf73e9))
### Alerts Volume
* [Weekly Trend](https://nonprod-log.gitlab.net/goto/4b20cce0-cb91-11ec-b3a6-472d0398dd6e)
* [Monthly Trend](https://nonprod-log.gitlab.net/goto/89e93020-cb91-11ec-b3a6-472d0398dd6e)
* [90 Days Trend by Service](https://nonprod-log.gitlab.net/goto/ed862250-cb91-11ec-b3a6-472d0398dd6e)
### 7 Day Issue Stats
* Oncall issues : **0**
* Access Request : **0**
* Change Issues : **8**
* Incident Issues : **18**
* CorrectiveAction Issues : **0**
#### Change Issues
* 2022-11-14T00:28:30Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8041+
* 2022-11-14T00:08:46Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8040+
* 2022-11-11T06:39:53Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8038+
* 2022-11-11T06:29:28Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8037+
* 2022-11-11T04:20:52Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8036+
* 2022-11-10T06:25:27Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8032+
* 2022-11-08T03:02:49Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8019+
* 2022-11-07T07:20:43Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8007+
#### Incident Issues
* 2022-11-10T22:03:49Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8035+ | reliability~3760142 | | `https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8035`
* 2022-11-10T18:17:50Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8034+ | reliability~3760139 | ~"Service::API" | `https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8034`
* 2022-11-10T15:55:55Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8033+ | reliability~3760141 | ~"Service::API" | `https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8033`
* 2022-11-09T18:36:29Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8030+ | reliability~3760142 | ~"Service::Prometheus" | `https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8030`
* 2022-11-09T14:56:21Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8029+ | reliability~3760141 | ~"Service::Logging" | `https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8029`
* 2022-11-09T14:33:39Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8028+ | reliability~3760141 | ~"Service::Pgbouncer" | `https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8028`
* 2022-11-09T13:30:17Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8027+ | reliability~3760141 | ~"Service::Customers" | `https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8027`
* 2022-11-09T08:05:49Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8025+ | reliability~3760140 | ~"Service::Gitaly" | `https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8025`
* 2022-11-09T05:04:56Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8024+ | reliability~3760141 | ~"Service::Sidekiq" | `https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8024`
* 2022-11-09T00:51:08Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8023+ | reliability~3760142 | ~"Service::Postgres" | `https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8023`
* 2022-11-08T14:21:40Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8022+ | reliability~3760141 | ~"Service::Gitaly" | `https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8022`
* 2022-11-08T14:09:23Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8021+ | reliability~3760140 | ~"Service::Pgbouncer" | `https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8021`
* 2022-11-08T04:45:08Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8020+ | reliability~3760141 | ~"Service::Prometheus" | `https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8020`
* 2022-11-08T02:58:28Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8018+ | reliability~3760141 | ~"Service::CustomersDot" | `https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8018`
* 2022-11-07T12:28:22Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8014+ | reliability~3760141 | ~"Service::Postgres" | `https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8014`
* 2022-11-07T12:01:24Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8013+ | reliability~3760142 | ~"Service::Logging" | `https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8013`
* 2022-11-07T08:56:18Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8010+ | reliability~3760141 | ~"Service::Git" | `https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8010`
* 2022-11-07T08:52:25Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8008+ | reliability~3760141 | ~"Service::Git" | `https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8008`
#### CorrectiveAction Issues
* 2022-11-08T22:53:09Z - https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/16799+
* 2022-11-08T22:36:30Z - https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/16798+
* 2022-11-07T22:06:37Z - https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/16788+
### Open Issue Stats
* [Oncall issues](https://gitlab.com/gitlab-com/infrastructure/issues?scope=all&utf8=%E2%9C%93&state=opened&label_name[]=oncall) : **2**
* [Change issues](https://gitlab.com/gitlab-com/production/issues?scope=all&utf8=%E2%9C%93&state=opened&label_name[]=change) : **28**
* [Incident issues](https://gitlab.com/gitlab-com/production/issues?scope=all&utf8=%E2%9C%93&state=opened&label_name[]=incident) : **10**
* [Access Request](https://gitlab.com/gitlab-com/infrastructure/issues?scope=all&utf8=%E2%9C%93&state=opened&label_name[]=access%20request) : **0**
* [CorrectiveAction](https://gitlab.com/gitlab-com/infrastructure/issues?scope=all&utf8=%E2%9C%93&state=opened&label_name[]=corrective%20action) : **94**
#### Open Change Issues
<details>
<summary>Show/Hide Table</summary>
| Created | Summary |
| ------- | ------- |
| 2022-11-14T00:28:30Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8041+ |
| 2022-11-14T00:08:46Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8040+ |
| 2022-11-11T06:39:53Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8038+ |
| 2022-11-11T06:29:28Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8037+ |
| 2022-11-11T04:20:52Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8036+ |
| 2022-11-10T06:25:27Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8032+ |
| 2022-11-01T17:39:50Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7976+ |
| 2022-11-01T10:03:04Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7969+ |
| 2022-11-01T00:52:47Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7967+ |
| 2022-11-01T00:52:32Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7966+ |
| 2022-10-28T17:02:10Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7942+ |
| 2022-10-25T13:23:59Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7922+ |
| 2022-10-18T07:03:19Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7893+ |
| 2022-10-18T06:45:39Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7892+ |
| 2022-10-14T03:26:01Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7877+ |
| 2022-10-11T15:17:53Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7862+ |
| 2022-10-07T22:14:21Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7848+ |
| 2022-10-06T07:05:45Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7840+ |
| 2022-10-05T01:45:36Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7836+ |
| 2022-10-05T00:00:51Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7834+ |
| 2022-09-23T15:35:33Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7793+ |
| 2022-09-02T09:37:55Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7681+ |
| 2022-08-25T14:11:48Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7653+ |
| 2022-08-17T19:40:56Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7615+ |
| 2022-08-16T16:43:58Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7607+ |
| 2022-07-22T14:23:23Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7497+ |
| 2022-06-06T15:55:01Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7206+ |
| 2022-05-23T16:22:23Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7117+ |
</details>
#### Open Incident Issues
<details>
<summary>Show/Hide Table</summary>
| Created | Summary |
| ------- | ------- |
| 2022-11-10T15:55:55Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8033+ |
| 2022-11-07T08:53:23Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8009+ |
| 2022-11-07T04:16:15Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8006+ |
| 2022-11-06T21:53:45Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8005+ |
| 2022-11-02T14:18:07Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7979+ |
| 2022-10-29T13:50:58Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7947+ |
| 2022-10-25T20:05:24Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7924+ |
</details>
#### Open Oncall Issues
<details>
<summary>Show/Hide Table</summary>
| Created | Summary |
| ------- | ------- |
| 2021-09-17T19:35:34Z | https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/14205+ |
| 2020-12-18T22:29:14Z | https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/12200+ |
</details>
#### Issues for Review during Incident Review Meeting
<details>
If there are any incidents you think would be good to review, please add them to the [Agenda](https://docs.google.com/document/d/1Llm9tXHC2dNt_eercRUUXlUyWmOVw00wmXWQQbWvv2c/edit?usp=sharing) for the next meeting.
</details>
issue