Weekly Reliability (SRE) Team Newsletter – On-call Period: 2022-08-23 - 2022-08-30

Announcements

Engineering Week in Review Highlights:

Team Updates

On-Call During This Period

Schedule Username
SRE 8-hour Americas Hendrik Meyer
SRE 8-hour Americas Nels Nelson
SRE 8-hour APAC Filipe Santos
SRE 8-hour EMEA Michal Wasilewski
SRE 8-hour EMEA Rehab Hassanein

PagerDuty Incidents

See the 1 week report for acknowledged PD pages (long-term trend)

Alerts Volume

7 Day Issue Stats

  • Oncall issues : 0
  • Access Request : 0
  • Change Issues : 11
  • Incident Issues : 14
  • CorrectiveAction Issues : 0

Change Issues

Incident Issues

CorrectiveAction Issues

Open Issue Stats

Open Change Issues

Show/Hide Table
Created Summary
2022-08-28T20:22:44Z Disable NUMA-hinted foreground page migrations ... (production#7660 - closed)
2022-08-26T14:28:22Z Enforce Cloudflare Authenticated Origin Pulls (production#7658 - closed)
2022-08-25T14:11:48Z [GPRD] Enable pg_wait_sampling in Production (production#7653 - closed)
2022-08-25T13:38:21Z 2022-08-25: Create new CI Runners projects in GCP (production#7652 - closed)
2022-08-25T09:46:49Z Download Jemalloc reports from web & api after ... (production#7651 - closed)
2022-08-23T10:33:31Z DRAFT: Create 4th k8s cluster on GSTG (production#7636 - closed)
2022-08-19T20:10:21Z Set cost factor for public shared runners (production#7628 - closed)
2022-08-19T12:51:52Z Update `GitLab Workflow VS Code Extension` inst... (production#7626 - closed)
2022-08-19T08:29:12Z 2022-08-19: [GSTG] Use turbo mode on restore co... (production#7624 - closed)
2022-08-17T19:40:56Z Move version.gitlab.com to new namespace, GCP p... (production#7615 - closed)
2022-08-16T16:43:58Z 2022-08-16: Increase max_client_conn for pgboun... (production#7607 - closed)
2022-08-16T14:31:39Z [GPRD] Install pg_stat_kcache package in Produc... (production#7606 - closed)
2022-08-15T10:36:05Z gstg-us-east1-b Zonal cluster rebuild (production#7598 - closed)
2022-08-12T15:04:02Z 2022-08-12: Opstrace Error Tracking Open Beta R... (production#7586 - closed)
2022-08-12T14:20:37Z 2022-09-21: GSTG Truncate the rest of CI tables... (production#7585 - closed)
2022-08-12T13:26:03Z Manually start migration for batch 1/2 of large... (production#7583 - closed)
2022-08-03T14:54:21Z https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7543+
2022-08-03T14:00:10Z Update `container_registry_import_created_befor... (production#7542 - closed)
2022-08-01T08:17:13Z 2022-08-16: GPRD Scale down the number of Patro... (production#7531 - closed)
2022-08-01T08:16:04Z https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7530+
2022-08-01T08:15:16Z 2022-08-14: GSTG Scale down the number of Patro... (production#7529 - closed)
2022-08-01T08:14:38Z 2022-08-13: GSTG Scale down the number of Patro... (production#7528 - closed)
2022-07-27T14:39:01Z Enable container_registry_phase_2_deny_list fea... (production#7518 - closed)
2022-07-25T16:23:58Z Disable container_registry_phase_2_deny_list fe... (production#7506 - closed)
2022-07-22T14:23:23Z 2022-07-22: Update home page url in staging fro... (production#7497 - closed)
2022-06-28T16:43:38Z Raise Tag Count Limit for Ongoing Phase 2 conta... (production#7343 - closed)
2022-06-27T14:16:42Z Reset Skipped Container Repository Imports for ... (production#7334 - closed)
2022-06-06T15:55:01Z Rollout usage of ci-gateway ILB for on shared r... (production#7206 - closed)
2022-05-23T16:22:23Z 2022-05-23: Update auth scopes to include servi... (production#7117 - closed)

Open Incident Issues

Show/Hide Table
Created Summary
2022-08-29T01:44:48Z 2022-08-29: RegistryServiceGarbagecollectorErro... (production#7661 - closed)
2022-08-25T22:34:22Z 2022-08-25: 401 errors downloading PyPi packages (production#7655 - closed)
2022-08-15T16:40:33Z 2022-08-15: Shared macOS Runners Failing (production#7601 - closed)
2022-08-09T17:31:57Z 2022-08-09: GCS slowness in zone us-east1-c aff... (production#7570 - closed)

Open Oncall Issues

Show/Hide Table
Created Summary
2021-09-17T19:35:34Z https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/14205+
2020-12-18T22:29:14Z https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/12200+

Issues for Review during Incident Review Meeting

If there are any incidents you think would be good to review, please add them to the Agenda for the next meeting.
Edited by ops-gitlab-net