Weekly Reliability (SRE) Team Newsletter – On-call Period: 2022-12-13 - 2022-12-20
Announcements
Engineering Week in Review Highlights:
Team Updates
On-Call During This Period
| Schedule | Username |
|---|---|
| SRE 8-hour Americas | Matt Smiley |
| SRE 8-hour Americas | Nels Nelson |
| SRE 8-hour APAC | Devin Sylva |
| SRE 8-hour APAC | Filipe Santos |
| SRE 8-hour APAC | Nick Duff |
| SRE 8-hour EMEA | Rehab Hassanein |
| SRE 8-hour EMEA | Igor Wiedler |
PagerDuty Incidents
See the 1 week report for acknowledged PD pages (long-term trend)
Alerts Volume
7 Day Issue Stats
- Oncall issues : 0
- Access Request : 0
- Change Issues : 11
- Incident Issues : 9
- CorrectiveAction Issues : 0
Change Issues
- 2022-12-16T09:48:46Z - [Production] Enqueue index_ci_builds_on_token_e... (production#8168 - closed)
- 2022-12-16T00:44:54Z - 2022-12-16: Upgrade HAProxy instances to t2d (production#8166 - closed)
- 2022-12-15T19:35:28Z - Container Registry: Apply post-deployment migra... (production#8165 - closed)
- 2022-12-15T19:18:54Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8164+
- 2022-12-15T15:46:51Z - Enable the "GitLab Migration" importer on stagi... (production#8163 - closed)
- 2022-12-15T15:19:22Z - 2023-01-03: Enable direct deb download for SaaS... (production#8162 - closed)
- 2022-12-13T14:27:37Z - Enable the "GitLab Migration" importer on .com ... (production#8158 - closed)
- 2022-12-13T14:02:02Z - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8157+
- 2022-12-13T05:41:29Z - 2022-12-13: Update server ssl cert for DR archi... (production#8154 - closed)
- 2022-12-13T03:34:38Z - 2022-12-13: Update server ssl cert for DR archi... (production#8153 - closed)
- 2022-12-12T12:45:48Z - 2022-12-15: [Private runners] enable Network Pe... (production#8151 - closed)
Incident Issues
- 2022-12-17T16:12:14Z - 2022-12-17: Abusive web requests trigger error ... (production#8170 - closed) | reliability~3760141 | ServiceWeb |
https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8170 - 2022-12-16T15:20:58Z - 2022-12-16: Grafana timeouts exceeding error SL... (production#8169 - closed) | reliability~3760141 | ServiceThanos |
https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8169 - 2022-12-16T05:36:48Z - 2022-12-16: QA Test Suites failing in gstg-cny ... (production#8167 - closed) | reliability~3760141 | ServiceWeb |
https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8167 - 2022-12-15T11:33:18Z - 2022-12-15: SidekiqServiceShardImportsApdexSLOV... (production#8161 - closed) | reliability~3760141 | ServiceSidekiq |
https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8161 - 2022-12-15T00:49:17Z - 2022-12-15: Intermittent 500 when visiting QA g... (production#8160 - closed) | reliability~3760141 | |
https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8160 - 2022-12-14T10:24:54Z - 2022-12-14: Sidekiq shard_urgent_cpu_bound satu... (production#8159 - closed) | reliability~3760140 | ServiceSidekiq |
https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8159 - 2022-12-13T13:30:12Z - 2022-12-13: Stall on patroni-main-2004-03 impac... (production#8156 - closed) | reliability~3760141 | ServicePostgres |
https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8156 - 2022-12-13T07:44:50Z - 2022-12-13: SidekiqServiceShardLowUrgencyCpuBou... (production#8155 - closed) | reliability~3760142 | ServiceSidekiq |
https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8155 - 2022-12-12T19:38:49Z - 2022-12-12: The grafana_google_lb SLI of the mo... (production#8152 - closed) | reliability~3760141 | ServiceGrafana |
https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8152
CorrectiveAction Issues
- 2022-12-12T15:01:17Z - https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/16964+
Open Issue Stats
- Oncall issues : 2
- Change issues : 24
- Incident issues : 7
- Access Request : 0
- CorrectiveAction : 91
Open Change Issues
Show/Hide Table
Open Incident Issues
Show/Hide Table
| Created | Summary |
|---|
Open Oncall Issues
Show/Hide Table
| Created | Summary |
|---|---|
| 2021-09-17T19:35:34Z | https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/14205+ |
| 2020-12-18T22:29:14Z | https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/12200+ |
Issues for Review during Incident Review Meeting
If there are any incidents you think would be good to review, please add them to the Agenda for the next meeting.
Edited by ops-gitlab-net