Weekly Reliability (SRE) Team Newsletter – On-call Period: 2022-06-21 - 2022-06-28

Announcements

Engineering Week in Review Highlights:

Team Updates

On-Call During This Period

Schedule Username
SRE 8-hour Americas Cameron McFarland
SRE 8-hour Americas Matt Smiley
SRE 8-hour Americas Marcel Chacon
SRE 8-hour APAC Pierre Guinoiseau
SRE 8-hour EMEA Rehab Hassanein
SRE 8-hour EMEA Igor Wiedler

PagerDuty Incidents

See the 1 week report for acknowledged PD pages (long-term trend)

Alerts Volume

7 Day Issue Stats

  • Oncall issues : 0
  • Access Request : 0
  • Change Issues : 11
  • Incident Issues : 30
  • CorrectiveAction Issues : 0

Change Issues

Incident Issues

CorrectiveAction Issues

Open Issue Stats

Open Change Issues

Show/Hide Table
Created Summary
2022-06-27T04:38:28Z 2022-06-27: Update remaining k8s node pools to use SSD
2022-06-27T02:56:00Z 2022-06-27: No-op update min number of nodes on all node pools
2022-06-24T01:09:12Z Draft: add stage label to fluentd-archiver pods
2022-06-23T21:07:49Z Draft: Enable application limit for incident management alerts
2022-06-23T03:31:48Z Delayed project authorization updates during bulk group share deletions
2022-06-17T18:50:05Z Setup GitLab for Jira app OAuth
2022-05-23T10:43:01Z 2022-07-02 05:00 UTC: Decompose GitLab.com's PostgreSQL Database into Main and CI
2022-05-18T06:17:33Z 2022-05-18: Enable inactive projects deletion on GitLab.com
2022-04-26T22:35:59Z Enable OSQuery on console hosts

Open Incident Issues

Show/Hide Table
Created Summary
2022-06-24T23:08:52Z 2022-06-24: Number of active Gitaly shards is low
2022-06-23T08:44:55Z 2022-06-23: RegistryServiceGarbagecollectorErrorSLOViolation
2022-06-22T15:55:51Z 2022-06-22: Workhorse apdex dips in canary
2022-06-22T13:33:05Z 2022-06-22: Elastic monitoring-cluster high disk allocation
2022-06-22T09:53:10Z 2022-06-22: alertmanager webhook integration failing with timeouts
2022-06-02T11:53:21Z 2022-06-02: Attempting to save changes to a large issue on ops.gitlab.net sometimes fails (403)

Open Oncall Issues

Show/Hide Table
Created Summary
2021-09-17T19:35:34Z Proposal: When an Incident is declared, output the latest changed feature flags into the incident issue
2020-12-18T22:29:14Z CI clones fail for repositories with a path ending in a period

Issues for Review during Incident Review Meeting

If there are any incidents you think would be good to review, please add them to the Agenda for the next meeting.
Edited by ops-gitlab-net