<!-- This issue was automatically generated by https://gitlab.com/gitlab-com/gl-infra/oncall-robot-assistant. --> <!-- Announcements common to all the Reliability (SRE) Teams should be placed in this section. --> # Announcements #### [Engineering Week in Review](https://docs.google.com/document/d/1JBdCl3MAOSdlgq3kzzRmtzTsFWsTIQ9iQg0RHhMht6E/edit) Highlights: <!-- Announcements for each individual SRE Team should be made in their respective sections below. --> # Team Updates <!-- xxYZzXcV --> --- # On-Call During This Period | Schedule | Username | | -------- | -------- | | SRE 8-hour Americas | Alex Hanselka | | SRE 8-hour Americas | Matt Smiley | | SRE 8-hour APAC | Devin Sylva | | SRE 8-hour EMEA | Ahmad Sherif | | SRE 8-hour EMEA | Alejandro Rodriguez | ## PagerDuty Incidents [See the 1 week report for acknowledged PD pages](https://nonprod-log.gitlab.net/app/dashboards#/view/dacc1d40-1c64-11ec-b8fd-b5d052b1f8cb?_g=(time:(from:'2022-09-20T01:00:00Z',to:'2022-09-27T01:00:00Z'),filters:!((query:(match_phrase:(type.keyword:pagerduty))),(query:(match_phrase:(status.keyword:triggered)))))) ([long-term trend](https://nonprod-log.gitlab.net/goto/a436702d57864666cd0c8867cfaf73e9)) ### Alerts Volume * [Weekly Trend](https://nonprod-log.gitlab.net/goto/4b20cce0-cb91-11ec-b3a6-472d0398dd6e) * [Monthly Trend](https://nonprod-log.gitlab.net/goto/89e93020-cb91-11ec-b3a6-472d0398dd6e) * [90 Days Trend by Service](https://nonprod-log.gitlab.net/goto/ed862250-cb91-11ec-b3a6-472d0398dd6e) ### 7 Day Issue Stats * Oncall issues : **0** * Access Request : **0** * Change Issues : **10** * Incident Issues : **22** * CorrectiveAction Issues : **0** #### Change Issues * 2022-09-23T15:35:33Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7793+ * 2022-09-23T02:05:57Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7790+ * 2022-09-22T03:09:47Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7785+ * 2022-09-21T17:20:28Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7783+ * 2022-09-21T15:20:39Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7782+ * 2022-09-21T15:18:22Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7781+ * 2022-09-21T15:16:37Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7780+ * 2022-09-21T08:16:10Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7776+ * 2022-09-21T08:16:05Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7775+ * 2022-09-20T16:16:49Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7770+ #### Incident Issues * 2022-09-24T19:13:00Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7795+ | reliability~3760141 | ~"Service::Logging" | `https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7795` * 2022-09-24T13:41:07Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7794+ | reliability~3760142 | ~"Service::GCP" | `https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7794` * 2022-09-23T12:09:49Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7792+ | reliability~3760141 | ~"Service::API" | `https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7792` * 2022-09-23T03:39:19Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7791+ | reliability~3760141 | ~"Service::GitLab Rails" | `https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7791` * 2022-09-22T23:44:30Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7789+ | reliability~3760141 | ~"Service::Monitoring-Other" | `https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7789` * 2022-09-22T22:01:34Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7788+ | reliability~3760141 | ~"Service::Web" | `https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7788` * 2022-09-22T13:59:15Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7787+ | reliability~3760141 | ~"Service::Logging" | `https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7787` * 2022-09-22T12:03:19Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7786+ | reliability~3760141 | ~"Service::Prometheus" | `https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7786` * 2022-09-21T17:47:24Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7784+ | reliability~3760140 | ~"Service::Gitaly" | `https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7784` * 2022-09-21T12:13:17Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7779+ | reliability~3760141 | | `https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7779` * 2022-09-21T10:04:26Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7778+ | reliability~3760141 | ~"Service::Thanos" | `https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7778` * 2022-09-21T06:57:57Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7774+ | reliability~3760141 | ~"Service::GitLab Rails" | `https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7774` * 2022-09-20T20:15:40Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7773+ | reliability~3760139 | ~"Service::Thanos" | `https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7773` * 2022-09-20T17:54:40Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7772+ | reliability~3760141 | ~"Service::Prometheus" | `https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7772` * 2022-09-20T17:54:14Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7771+ | reliability~3760141 | ~"Service::Logging" | `https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7771` * 2022-09-20T15:01:24Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7769+ | reliability~3760140 | | `https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7769` * 2022-09-20T13:21:38Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7768+ | reliability~3760142 | ~"Service::CI Runners" | `https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7768` * 2022-09-20T10:48:34Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7767+ | reliability~3760142 | ~"Service::Windows CI Runner" | `https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7767` * 2022-09-20T05:26:25Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7766+ | reliability~3760141 | ~"Service::Postgres" | `https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7766` * 2022-09-19T22:20:13Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7765+ | reliability~3760141 | ~"Service::GitLab Rails" | `https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7765` * 2022-09-19T11:59:06Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7762+ | reliability~3760141 | | `https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7762` * 2022-09-19T11:50:02Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7761+ | reliability~3760141 | ~"Service::Chef" | `https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7761` #### CorrectiveAction Issues * 2022-09-20T19:27:57Z - https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/16431+ * 2022-09-20T19:06:16Z - https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/16430+ * 2022-09-19T11:55:56Z - https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/16416+ ### Open Issue Stats * [Oncall issues](https://gitlab.com/gitlab-com/infrastructure/issues?scope=all&utf8=%E2%9C%93&state=opened&label_name[]=oncall) : **2** * [Change issues](https://gitlab.com/gitlab-com/production/issues?scope=all&utf8=%E2%9C%93&state=opened&label_name[]=change) : **23** * [Incident issues](https://gitlab.com/gitlab-com/production/issues?scope=all&utf8=%E2%9C%93&state=opened&label_name[]=incident) : **10** * [Access Request](https://gitlab.com/gitlab-com/infrastructure/issues?scope=all&utf8=%E2%9C%93&state=opened&label_name[]=access%20request) : **0** * [CorrectiveAction](https://gitlab.com/gitlab-com/infrastructure/issues?scope=all&utf8=%E2%9C%93&state=opened&label_name[]=corrective%20action) : **95** #### Open Change Issues <details> <summary>Show/Hide Table</summary> | Created | Summary | | ------- | ------- | | 2022-09-23T15:35:33Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7793+ | | 2022-09-21T15:18:22Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7781+ | | 2022-09-21T15:16:37Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7780+ | | 2022-09-21T08:16:10Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7776+ | | 2022-09-21T08:16:05Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7775+ | | 2022-09-20T16:16:49Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7770+ | | 2022-09-19T06:30:03Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7759+ | | 2022-09-19T06:15:09Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7758+ | | 2022-09-15T14:50:38Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7749+ | | 2022-09-14T03:37:44Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7732+ | | 2022-09-02T09:37:55Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7681+ | | 2022-08-25T14:11:48Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7653+ | | 2022-08-17T19:40:56Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7615+ | | 2022-08-16T16:43:58Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7607+ | | 2022-08-15T10:36:05Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7598+ | | 2022-08-12T15:04:02Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7586+ | | 2022-08-01T08:17:13Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7531+ | | 2022-08-01T08:16:04Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7530+ | | 2022-07-22T14:23:23Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7497+ | | 2022-06-28T16:43:38Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7343+ | | 2022-06-27T14:16:42Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7334+ | | 2022-06-06T15:55:01Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7206+ | | 2022-05-23T16:22:23Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7117+ | </details> #### Open Incident Issues <details> <summary>Show/Hide Table</summary> | Created | Summary | | ------- | ------- | | 2022-09-22T22:01:34Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7788+ | | 2022-09-20T17:54:40Z | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7772+ | </details> #### Open Oncall Issues <details> <summary>Show/Hide Table</summary> | Created | Summary | | ------- | ------- | | 2021-09-17T19:35:34Z | https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/14205+ | | 2020-12-18T22:29:14Z | https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/12200+ | </details> #### Issues for Review during Incident Review Meeting <details> If there are any incidents you think would be good to review, please add them to the [Agenda](https://docs.google.com/document/d/1Llm9tXHC2dNt_eercRUUXlUyWmOVw00wmXWQQbWvv2c/edit?usp=sharing) for the next meeting. </details>