Weekly Reliability (SRE) Team Newsletter – On-call Period: 2023-05-16 - 2023-05-23

Announcements

Engineering Week in Review Highlights:

Team Updates

On-Call During This Period

Schedule Username
SRE 8-hour Americas Alejandro Rodríguez
SRE 8-hour Americas Marcel Chacon
SRE 8-hour APAC Gonzalo Servat
SRE 8-hour APAC Furhan Shabir
SRE 8-hour EMEA Ahmad Sherif
SRE 8-hour EMEA Steve Azzopardi

PagerDuty Incidents

See the 1 week report for acknowledged PD pages (long-term trend)

Alerts Volume

7 Day Issue Stats

  • Oncall issues : 0
  • Access Request : 0
  • Change Issues : 11
  • Incident Issues : 21
  • CorrectiveAction Issues : 0

Change Issues

Incident Issues

CorrectiveAction Issues

Open Issue Stats

Open Change Issues

Show/Hide Table
Created Summary
2023-05-19T20:50:31Z [CR] [gprd] Switch 100% to new HAProxy 2.6 (production#14475 - closed)
2023-05-19T18:22:55Z [CR] [gprd] Switch to new HAProxy 2.6 CI intern... (production#14473 - closed)
2023-05-19T18:21:29Z [CR] [gprd] Switch to new HAProxy 2.6 internal ... (production#14472 - closed)
2023-05-19T18:20:14Z [CR] [gprd] Send a small percentage of traffic ... (production#14471 - closed)
2023-05-18T18:24:29Z https://gitlab.com/gitlab-com/gl-infra/production/-/issues/14456+
2023-05-18T10:34:27Z [GPRD] Complete ci_builds partitioning - 2nd at... (production#14453 - closed)
2023-05-18T08:07:10Z [GPRD] Install pg_stat_kcache package in DR pos... (production#14449 - closed)
2023-05-17T08:23:17Z [GPRD] [2023-08-26] - Upgrade PostgreSQL to PG1... (production#14403 - closed)
2023-05-17T08:20:25Z [GPRD] [2023-06-24 14:00 UTC -19:00 UTC] - Upgr... (production#14402 - closed)
2023-05-16T05:50:31Z [GPRD] Recreate invalid index_vulnerability_rea... (production#14368 - closed)
2023-05-16T00:39:13Z 2023-05-16: test that Redis and Sentinel start ... (production#14365 - closed)
2023-05-05T13:37:54Z [GPRD] [2023-06-10 14:00 UTC - 19:00 UTC] - Upg... (production#11375 - closed)
2023-05-04T02:19:45Z 2023-05-04: Remove admin rights from ops-gitlab... (production#11074 - closed)
2023-05-02T03:55:12Z [GPRD] [2023-05-27 03:00 UTC] CR - Hardware upg... (production#10694 - closed)
2023-04-28T13:13:26Z [GSTG][2023-05-31T14:00Z] Upgrade PostgreSQL to... (production#9630 - closed)
2023-04-26T07:56:53Z [gprd] Disable `security_policy_global_group_ap... (production#9130 - closed)
2023-04-21T18:04:19Z 2023-05-23: DEV pgupgrade to update database to... (production#8779 - closed)
2023-04-19T12:27:25Z [gstg] Disable `security_policy_global_group_ap... (production#8750 - closed)
2023-04-05T17:23:13Z [2023-04-19 23:45 UTC][GSTG] Patroni Main Clust... (production#8678 - closed)
2023-04-03T14:55:16Z [GPRD] Prepare for: Execute logical replication... (production#8653 - closed)
2023-03-28T16:11:57Z [GPRD] Execute logical replication and upgrade ... (production#8611 - closed)
2023-02-22T00:27:16Z 2023-02-22: migrate grafana thanos datasource (production#8438 - closed)
2023-02-02T16:44:37Z https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8352+
2022-12-22T03:57:59Z 2022-12-22: Add registry expiration policies to... (production#8186 - closed)
2022-11-14T23:01:34Z Change the schedule of automatic database reind... (production#8048 - closed)
2022-10-18T07:03:19Z 2022-10-18: [GPRD] Use turbo mode on restore co... (production#7893 - closed)
2022-10-18T06:45:39Z 2022-10-18: [GPRD] Use turbo mode on restore co... (production#7892 - closed)
2022-10-06T07:05:45Z Disable fastupdate on merge_requests GIN indexes (production#7840 - closed)
2022-10-05T00:00:51Z Reduce io-threads on redis-persistent in staging (production#7834 - closed)
2022-09-02T09:37:55Z Removal of sigin-Page text for staging and .com (production#7681 - closed)
2022-08-25T14:11:48Z [GPRD] Enable pg_wait_sampling in Production (production#7653 - closed)
2022-08-16T16:43:58Z 2022-08-16: Increase max_client_conn for pgboun... (production#7607 - closed)
2022-06-06T15:55:01Z Rollout usage of ci-gateway ILB for on shared r... (production#7206 - closed)
2022-05-23T16:22:23Z 2022-05-23: Update auth scopes to include servi... (production#7117 - closed)

Open Incident Issues

Show/Hide Table
Created Summary

Open Oncall Issues

Show/Hide Table
Created Summary
2021-09-17T19:35:34Z https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/14205+
2020-12-18T22:29:14Z https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/12200+

Issues for Review during Incident Review Meeting

If there are any incidents you think would be good to review, please add them to the Agenda for the next meeting.
Edited by ops-gitlab-net