Weekly Reliability (SRE) Team Newsletter – On-call Period: 2022-08-09 - 2022-08-16

Announcements

Engineering Week in Review Highlights:

Team Updates

On-Call During This Period

Schedule Username
SRE 8-hour Americas Alex Hanselka
SRE 8-hour Americas Matt Smiley
SRE 8-hour APAC Devin Sylva
SRE 8-hour APAC Pierre Guinoiseau
SRE 8-hour EMEA Alejandro Rodriguez
SRE 8-hour EMEA Steve Azzopardi

PagerDuty Incidents

See the 1 week report for acknowledged PD pages (long-term trend)

Alerts Volume

7 Day Issue Stats

  • Oncall issues : 0
  • Access Request : 0
  • Change Issues : 13
  • Incident Issues : 16
  • CorrectiveAction Issues : 1

Change Issues

Incident Issues

CorrectiveAction Issues

Open Issue Stats

Open Change Issues

Show/Hide Table
Created Summary
2022-08-15T03:52:53Z 2022-08-15: GSTG - Migrate TF module for Patron... (production#7595 - closed)
2022-08-12T16:07:57Z Manually start migration for batch 2/2 of large... (production#7588 - closed)
2022-08-12T15:04:02Z 2022-08-12: Opstrace Error Tracking Open Beta R... (production#7586 - closed)
2022-08-12T14:20:37Z 2022-09-21: GSTG Truncate the rest of CI tables... (production#7585 - closed)
2022-08-12T13:26:03Z Manually start migration for batch 1/2 of large... (production#7583 - closed)
2022-08-12T05:18:48Z [08/16/2022 - 00:00 UTC] - GSTG - Disable post ... (production#7582 - closed)
2022-08-11T09:42:48Z 2022-08-15: Reduce the default TTL for Rails.cache (production#7580 - closed)
2022-08-10T15:10:06Z [08/17/2022 - 00:00 UTC] - Patroni Clusters OS ... (production#7577 - closed)
2022-08-10T08:58:23Z [production] Enable `always_async_project_autho... (production#7574 - closed)
2022-08-04T19:10:13Z Re-import projects into project_build_artifacts... (production#7553 - closed)
2022-08-04T15:39:55Z [staging] Update tokens and URL to enable secre... (production#7549 - closed)
2022-08-03T14:54:21Z https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7543+
2022-08-03T14:00:10Z Update `container_registry_import_created_befor... (production#7542 - closed)
2022-08-01T08:17:13Z 2022-08-16: GPRD Scale down the number of Patro... (production#7531 - closed)
2022-08-01T08:16:04Z https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7530+
2022-08-01T08:15:16Z 2022-08-14: GSTG Scale down the number of Patro... (production#7529 - closed)
2022-08-01T08:14:38Z 2022-08-13: GSTG Scale down the number of Patro... (production#7528 - closed)
2022-07-27T14:39:01Z Enable container_registry_phase_2_deny_list fea... (production#7518 - closed)
2022-07-25T16:23:58Z Disable container_registry_phase_2_deny_list fe... (production#7506 - closed)
2022-07-22T14:23:23Z 2022-07-22: Update home page url in staging fro... (production#7497 - closed)
2022-06-28T16:43:38Z Raise Tag Count Limit for Ongoing Phase 2 conta... (production#7343 - closed)
2022-06-27T14:16:42Z Reset Skipped Container Repository Imports for ... (production#7334 - closed)
2022-06-06T15:55:01Z Rollout usage of ci-gateway ILB for on shared r... (production#7206 - closed)
2022-05-23T16:22:23Z 2022-05-23: Update auth scopes to include servi... (production#7117 - closed)
2022-05-18T06:17:33Z https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7086+

Open Incident Issues

Show/Hide Table
Created Summary
2022-08-09T17:31:57Z 2022-08-09: GCS slowness in zone us-east1-c aff... (production#7570 - closed)
2022-08-08T12:16:27Z 2022-08-08: pgbouncer_client_conn_primary satur... (production#7565 - closed)

Open Oncall Issues

Show/Hide Table
Created Summary
2021-09-17T19:35:34Z https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/14205+
2020-12-18T22:29:14Z https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/12200+

Issues for Review during Incident Review Meeting

If there are any incidents you think would be good to review, please add them to the Agenda for the next meeting.
Edited by ops-gitlab-net